Sep 16 18:42:26.441780 ip-10-0-130-134 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:42:26.441791 ip-10-0-130-134 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:42:26.441799 ip-10-0-130-134 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:42:26.442096 ip-10-0-130-134 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:42:35.849627 ip-10-0-130-134 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:42:35.849644 ip-10-0-130-134 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 31065344fe4942d792603a9528373759 --
Sep 16 18:44:09.698179 ip-10-0-130-134 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:44:10.129261 ip-10-0-130-134 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:10.129261 ip-10-0-130-134 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:44:10.129261 ip-10-0-130-134 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:10.129261 ip-10-0-130-134 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:44:10.129261 ip-10-0-130-134 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:10.130255 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.130123    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:44:10.137388 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137346    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:10.137388 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137381    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:10.137388 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137387    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:10.137388 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137392    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:10.137388 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137399    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137403    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137407    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137412    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137417    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137421    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137425    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137430    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137433    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137437    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137441    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137444    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137448    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137452    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137456    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137461    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137465    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137468    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137472    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137476    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:10.137640 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137480    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137484    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137488    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137492    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137496    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137500    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137504    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137508    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137512    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137517    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137522    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137526    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137529    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137534    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137538    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137542    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137546    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137550    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137554    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137558    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:10.138136 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137562    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137565    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137571    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137578    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137582    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137586    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137591    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137596    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137601    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137605    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137609    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137612    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137616    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137620    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137624    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137627    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137631    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137635    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137639    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137644    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:10.138627 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137648    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137652    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137655    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137659    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137663    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137667    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137673    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137676    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137680    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137684    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137688    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137691    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137695    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137699    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137702    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137706    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137709    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137713    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137717    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137725    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:10.139135 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137730    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137734    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137738    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137741    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137745    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137749    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137753    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137756    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137761    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137765    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137771    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137775    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137778    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137782    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:10.139656 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.137786    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139861    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139877    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139883    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139888    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139892    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139897    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139900    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139904    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139908    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139912    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139916    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139919    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139923    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139927    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139930    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139934    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139938    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139943    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:10.140021 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139947    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139950    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139954    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139958    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139961    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139965    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139968    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139972    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139976    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139979    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139984    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139987    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139993    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.139998    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140002    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140007    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140011    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140016    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140021    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140025    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:10.140517 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140030    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140034    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140039    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140043    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140047    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140051    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140055    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140059    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140063    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140067    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140071    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140074    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140079    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140083    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140087    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140090    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140094    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140099    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140102    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140105    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:10.141033 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140109    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140113    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140116    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140120    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140124    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140128    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140132    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140138    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140141    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140145    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140149    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140152    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140156    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140161    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140164    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140168    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140173    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140177    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140181    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:10.141527 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140185    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140189    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140193    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140196    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140200    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140204    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140209    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140213    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140218    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140225    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140230    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140234    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140238    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140241    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140245    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140249    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140253    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140257    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140262    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140265    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:10.142018 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140271    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140278    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140912    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140921    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140926    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140930    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140934    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140938    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140942    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140946    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140950    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140954    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140957    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140961    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140965    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140969    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140973    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140976    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140981    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:10.142507 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140985    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140989    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140993    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.140998    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141001    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141006    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141010    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141014    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141018    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141024    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141029    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141033    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141038    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141042    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141047    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141050    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141054    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141057    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141061    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:10.142990 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141065    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141069    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141074    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141077    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141082    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141085    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141089    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141092    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141096    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141100    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141103    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141107    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141111    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141116    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141119    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141123    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141127    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141131    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141135    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141139    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:10.143465 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141143    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141147    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141150    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141153    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141157    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141161    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141165    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141170    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141174    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141180    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141184    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141188    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141192    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141196    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141199    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141203    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141206    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141210    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141214    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141218    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:10.143979 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141222    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141228    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141231    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141236    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141241    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141246    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141251    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141255    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141259    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141262    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141266    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141270    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141273    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141277    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141281    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141284    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141288    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141292    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141295    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141299    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:10.144503 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141303    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141307    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.141310    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141422    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141481    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141494    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141501    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141508    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141513    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141519    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141525    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141531    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141537    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141542    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141548    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141553    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141558    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141562    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141567    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141573    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141577    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141584    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141588    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:44:10.145016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141593    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141597    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141602    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141608    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141612    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141617    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141622    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141627    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141632    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141636    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141641    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141646    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141652    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141656    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141661    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141665    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141670    2576 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141674    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141686    2576 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141690    2576 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141695    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141700    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141704    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141710    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:44:10.145625 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141714    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141719    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141725    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141729    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141734    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141747    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141752    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141756    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141761    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141765    2576 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141771    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141776    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141781    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141786    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141791    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141796    2576 flags.go:64] FLAG: --help="false"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141800    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141805    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141809    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141814    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141823    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141829    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141849    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:44:10.146231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141854    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141859    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141863    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141867    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141873    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141878    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141882    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141887    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141891    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141896    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141901    2576 flags.go:64] FLAG: --lock-file=""
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141908    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141912    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141917    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141925    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141939    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141944    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141948    2576 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141953    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141958    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141962    2576 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141966    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141973    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141978    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141984    2576 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:44:10.146825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141989    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141994    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.141998    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142003    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142009    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142013    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142018    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142031    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142036    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142040    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142045    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142049    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142058    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142062    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142068    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142072    2576 flags.go:64] FLAG: --port="10250"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142077    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142081    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07c924054800f22b8"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142086    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142092    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142098    2576 flags.go:64] FLAG: --register-node="true"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142102    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142108    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142120    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:44:10.147466 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142125    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142129    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142135    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142140    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142145    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142150    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142154    2576 flags.go:64] FLAG: --runonce="false"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142158    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142163    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142168    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142172    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142177    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142183    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142188    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142193    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142198    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142202    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142206    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142211    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142215    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142220    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142224    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142232    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142237    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142242    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:44:10.148125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142254    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142258    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142265    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142270    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142274    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142279    2576 flags.go:64] FLAG: --v="2"
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142284    2576 flags.go:64] FLAG: --version="false"
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142292    2576 flags.go:64] FLAG: --vmodule=""
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142297    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.142303    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142482    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142489    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142495    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142500    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142504    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142508    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142513    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142517    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142521    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142528    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142531    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:10.148731 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142536    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142540    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142544    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142548    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142552    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142556    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142560    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142566    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142570    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142574    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142577    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142581    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142585    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142590    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142594    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142598    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142602    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142606    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142610    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142614    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:10.149267 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142618    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142622    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142626    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142630    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142633    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142638    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142642    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142646    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142650    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142654    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142659    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142663    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142667    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142671    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142675    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142678    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142682    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142686    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142692    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142697    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:10.149758 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142701    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142705    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142709    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142713    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142717    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142723    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142727    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142730    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142734    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142738    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142742    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142746    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142751    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142755    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142758    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142764    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142769    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142773    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142777    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:10.150273 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142781    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142785    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142789    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142795    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142799    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142804    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142808    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142811    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142815    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142822    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142827    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142831    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142848    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142852    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142856    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142860    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142863    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142867    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142876    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142880    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:10.150782 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142883    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:10.151326 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142887    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:10.151326 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142891    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:10.151326 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142896    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:10.151326 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142900    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:10.151326 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142905    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:10.151326 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142910    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:10.151326 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142913    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:10.151326 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.142917    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:10.151326 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.143589    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:10.152000 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.151973    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:44:10.152034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.152000    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:44:10.152034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.152025    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:44:10.152094 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.152034    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:44:10.152122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152095    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:10.152122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152100    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:10.152122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152105    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:10.152122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152110    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:10.152122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152113    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:10.152122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152116    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:10.152122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152119    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:10.152122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152122    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:10.152122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152126    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152129    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152132    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152135    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152138    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152141    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152144    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152147    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152149    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152152    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152156    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152160    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152163    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152166    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152169    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152172    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152175    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152178    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152181    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:10.152342 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152184    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152186    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152190    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152193    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152195    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152198    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152200    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152203    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152205    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152208    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152211    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152213    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152215    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152219    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152222    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152225    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152228    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152230    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152233    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152236    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:10.152819 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152239    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152242    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152244    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152247    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152250    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152252    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152255    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152258    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152262    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152264    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152267    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152270    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152272    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152275    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152278    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152281    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152284    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152286    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152289    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152291    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:10.153417 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152294    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152296    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152300    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152303    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152306    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152309    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152312    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152315    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152317    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152320    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152323    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152326    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152329    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152331    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152334    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152336    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152339    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152341    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152345    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:10.153913 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152349    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152352    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152355    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152358    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152360    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152363    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152366    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152368    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152371    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152374    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152377    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152380    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152382    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.152388    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:10.154374 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152554    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152561    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152565    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152568    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152572    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152575    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152579    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152582    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152585    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152588    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152591    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152595    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152599    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152604    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152608    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152611    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152614    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152617    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152620    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:10.154729 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152623    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152626    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152629    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152632    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152634    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152637    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152640    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152642    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152645    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152653    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152657    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152659    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152662    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152665    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152668    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152671    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152673    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152676    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152678    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:10.155205 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152682    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152684    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152687    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152690    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152692    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152695    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152698    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152700    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152703    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152706    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152708    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152712    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152714    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152717    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152720    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152722    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152725    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152727    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152730    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152733    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:10.155668 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152736    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152738    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152741    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152750    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152753    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152756    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152758    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152761    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152764    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152767    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152771    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152775    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152779    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152782    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152785    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152788    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152791    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152794    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152797    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152799    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:10.156232 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152802    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152805    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152808    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152810    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152813    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152816    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152818    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152821    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152824    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152827    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152832    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152851    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152854    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152857    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152860    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152863    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152878    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152881    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152883    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152886    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:10.156725 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:10.152889    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:10.157285 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.152895    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:10.157285 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.153065    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:44:10.157285 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.155707    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:44:10.157285 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.156927    2576 server.go:1019] "Starting client certificate rotation"
Sep 16 18:44:10.157285 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.157049    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:10.157285 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.157094    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:10.160955 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.160935    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:10.165691 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.165661    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:10.176395 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.176365    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:44:10.181090 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.181068    2576 log.go:25] "Validated CRI v1 image API"
Sep 16 18:44:10.182114 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.182084    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:44:10.185854 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.185797    2576 fs.go:135] Filesystem UUIDs: map[502ab848-42c6-4a5b-8b9c-13989802ec05:/dev/nvme0n1p3 65692853-6ec3-43b2-88ac-7966fb8cdef8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 18:44:10.185984 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.185854    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:44:10.192748 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.192432    2576 manager.go:216] Machine: {Timestamp:2026-09-16 18:44:10.190959492 +0000 UTC m=+0.384823155 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098730 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24bee2b2794ef531cb07729b8d0146 SystemUUID:ec24bee2-b279-4ef5-31cb-07729b8d0146 BootID:31065344-fe49-42d7-9260-3a9528373759 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:28:91:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:28:91:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:10:08:fa:b1:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:44:10.192748 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.192725    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:44:10.192959 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.192705    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:10.192959 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.192889    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:44:10.196882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.196814    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:44:10.197098 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.196885    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:44:10.197880 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.197866    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:44:10.197916 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.197882    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:44:10.197916 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.197890    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:10.198713 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.198699    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:10.200043 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.200027    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:44:10.204815 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.204787    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:10.205177 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.205162    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:44:10.205232 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.205218    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:44:10.205232 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.205230    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:44:10.205315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.205242    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:44:10.205315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.205257    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:44:10.205315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.205267    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:44:10.205757 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.205744    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:10.205826 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.205765    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:10.208509 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.208485    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:44:10.211739 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.211715    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:44:10.211811 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.211754    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:44:10.213821 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.213805    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:44:10.213821 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.213823    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:44:10.213821 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.213829    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:44:10.213999 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.213849    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:44:10.213999 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.213861    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:44:10.214094 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.213867    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:44:10.214457 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.214435    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:44:10.214457 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.214458    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:44:10.214576 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.214473    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:44:10.214576 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.214484    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:44:10.214576 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.214515    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:44:10.214576 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.214525    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:44:10.215490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.215468    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:44:10.215490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.215492    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:44:10.218476 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.218440    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:44:10.218476 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.218436    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:44:10.218673 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.218541    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:44:10.220536 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.220523    2576 server.go:1268] "Started kubelet"
Sep 16 18:44:10.220644 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.220606    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:44:10.220755 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.220615    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:44:10.220755 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.220690    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:44:10.221591 ip-10-0-130-134 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:44:10.221722 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.221683    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:44:10.222411 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.222393    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:44:10.225851 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.225817    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:44:10.229497 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.229475    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:44:10.229497 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.229497    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:10.229671 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.229613    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:10.230440 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230421    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:44:10.230440 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230442    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:44:10.230583 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230523    2576 factory.go:55] Registering systemd factory
Sep 16 18:44:10.230583 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230542    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:44:10.230583 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230563    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:44:10.230703 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230685    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:44:10.230703 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230693    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:44:10.230887 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230872    2576 factory.go:153] Registering CRI-O factory
Sep 16 18:44:10.230887 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230888    2576 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:44:10.231000 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230933    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:44:10.231000 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230950    2576 factory.go:103] Registering Raw factory
Sep 16 18:44:10.231000 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.230962    2576 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:44:10.231340 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.231320    2576 manager.go:349] Starting recovery of all containers
Sep 16 18:44:10.238447 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.238394    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:44:10.239011 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.238980    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:44:10.239487 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.238187    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-134.ec2.internal.18d5e164da069af3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-134.ec2.internal,UID:ip-10-0-130-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-134.ec2.internal,},FirstTimestamp:2026-09-16 18:44:10.220485363 +0000 UTC m=+0.414349026,LastTimestamp:2026-09-16 18:44:10.220485363 +0000 UTC m=+0.414349026,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-134.ec2.internal,}"
Sep 16 18:44:10.243737 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.243711    2576 manager.go:354] Recovery completed
Sep 16 18:44:10.245548 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.245522    2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 18:44:10.249001 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.248975    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5kn24"
Sep 16 18:44:10.250652 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.250639    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:10.253169 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.253152    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:10.253219 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.253190    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:10.253219 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.253202    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:10.253856 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.253822    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:44:10.253856 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.253856    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:44:10.253966 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.253874    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:10.254008 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.253997    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:10.254589 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.254562    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5kn24"
Sep 16 18:44:10.254760 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.254739    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:10.254870 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.254769    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:10.254870 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.254779    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:10.256144 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.256129    2576 policy_none.go:49] "None policy: Start"
Sep 16 18:44:10.256144 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.256146    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:44:10.256239 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.256157    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:44:10.258089 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.258077    2576 policy_none.go:47] "Start"
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.301575    2576 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.301622    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.301633    2576 server.go:87] "Starting device plugin registration server"
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.301967    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.301978    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.302087    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.302168    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.302177    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.302799    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:44:10.306807 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.302862    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:10.327307 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.327259    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:44:10.328501 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.328480    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:44:10.328564 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.328557    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:44:10.328603 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.328593    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:44:10.328659 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.328644    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:44:10.331133 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.331113    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:10.402354 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.402240    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:10.403297 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.403280    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:10.403399 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.403311    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:10.403399 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.403321    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:10.403399 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.403343    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.412937 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.412906    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.413007 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.412943    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-134.ec2.internal\": node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:10.429431 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.429382    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"]
Sep 16 18:44:10.429543 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.429482    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:10.431104 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.431083    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:10.431104 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.431094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a967a84e42dca6d85c597cf3068ce6a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"a967a84e42dca6d85c597cf3068ce6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.431250 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.431113    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:10.431250 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.431123    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:10.431250 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.431126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a967a84e42dca6d85c597cf3068ce6a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"a967a84e42dca6d85c597cf3068ce6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.433538 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.433526    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:10.433660 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.433643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.433709 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.433681    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:10.434420 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.434399    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:10.434494 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.434399    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:10.434494 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.434431    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:10.434494 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.434443    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:10.434494 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.434467    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:10.434494 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.434481    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:10.434851 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.434824    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:10.436806 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.436789    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.436907 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.436821    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:10.437700 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.437684    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:10.437796 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.437713    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:10.437796 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.437726    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:10.453521 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.453494    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-134.ec2.internal\" not found" node="ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.458259 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.458242    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-134.ec2.internal\" not found" node="ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.531964 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.531927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a967a84e42dca6d85c597cf3068ce6a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"a967a84e42dca6d85c597cf3068ce6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.531964 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.531968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b26e3b59ca8b326d0e3c11878453793-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" (UID: \"4b26e3b59ca8b326d0e3c11878453793\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.532125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.531986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a967a84e42dca6d85c597cf3068ce6a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"a967a84e42dca6d85c597cf3068ce6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.532125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.532035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a967a84e42dca6d85c597cf3068ce6a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"a967a84e42dca6d85c597cf3068ce6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.532125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.532037    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a967a84e42dca6d85c597cf3068ce6a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"a967a84e42dca6d85c597cf3068ce6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.535031 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.535013    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:10.632247 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.632214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b26e3b59ca8b326d0e3c11878453793-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" (UID: \"4b26e3b59ca8b326d0e3c11878453793\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.632358 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.632276    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b26e3b59ca8b326d0e3c11878453793-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" (UID: \"4b26e3b59ca8b326d0e3c11878453793\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.635520 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.635498    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:10.735700 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.735655    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:10.756376 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.756340    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.760665 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:10.760641    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:10.836926 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.836883    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:10.937923 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:10.937885    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:11.039084 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.038971    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:11.139170 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.139111    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found"
Sep 16 18:44:11.156980 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.156935    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:44:11.157157 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.157137    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:11.173248 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.173212    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:11.205811 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.205761    2576 apiserver.go:52] "Watching apiserver"
Sep 16 18:44:11.214349 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.214317    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:44:11.214728 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.214707    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q","openshift-cluster-node-tuning-operator/tuned-lx59p","openshift-multus/multus-gckt2","openshift-network-diagnostics/network-check-target-gglxk","openshift-ovn-kubernetes/ovnkube-node-4x5hs","kube-system/global-pull-secret-syncer-49tdd","kube-system/konnectivity-agent-hs9zv","openshift-dns/node-resolver-6rmzs","openshift-image-registry/node-ca-x4m4g","openshift-multus/multus-additional-cni-plugins-m8fxx","openshift-multus/network-metrics-daemon-fzv4f","openshift-network-operator/iptables-alerter-x4vvb"]
Sep 16 18:44:11.217584 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.217560    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.221297 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.221259    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.221478 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.221419    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:44:11.221478 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.221427    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ppz9c\""
Sep 16 18:44:11.221612 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.221525    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:44:11.221612 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.221419    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:44:11.223854 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.223815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.224370 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.224338    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:11.224370 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.224364    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zfq85\""
Sep 16 18:44:11.224715 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.224699    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:11.225765 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.225749    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:44:11.225940 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.225923    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:44:11.226195 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.226178    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:11.226304 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.226282    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:11.226977 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.226960    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qt468\""
Sep 16 18:44:11.227045 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.226962    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:44:11.227045 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.227031    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:44:11.228679 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.228662    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.230702 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.230683    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:11.230702 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.230698    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:11.231618 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.231434    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:44:11.231618 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.231460    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:44:11.231618 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.231495    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b52d8\""
Sep 16 18:44:11.231826 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.231785    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:44:11.231826 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.231803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.231944 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.231889    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:11.232056 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.232042    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:44:11.232239 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.232226    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:44:11.232461 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.232448    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:44:11.234473 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.234451    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:11.236456 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.236436    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:44:11.237082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.237065    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.237285 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.237264    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tdrlg\""
Sep 16 18:44:11.237371 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.237309    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:44:11.238710 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-daemon-config\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.238796 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-cni-dir\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.238796 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-registration-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.238796 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-sysconfig\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.238967 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-host\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.238967 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:11.238967 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-var-lib-openvswitch\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.238967 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238898    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-run-openvswitch\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.238967 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c18ed6a5-1b44-4261-a9cc-c5e78164115e-ovnkube-script-lib\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.239206 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.238989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-device-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.239206 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-systemd-units\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.239206 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-cni-bin\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.239206 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c18ed6a5-1b44-4261-a9cc-c5e78164115e-env-overrides\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.239206 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-cni-binary-copy\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.239206 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-kubernetes\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.239490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-lib-modules\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.239490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-run-ovn\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.239490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-node-log\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.239490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c18ed6a5-1b44-4261-a9cc-c5e78164115e-ovn-node-metrics-cert\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.239490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.239490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-modprobe-d\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.239490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239387    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:44:11.239490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-tuned\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.239490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239494    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239498    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-run-k8s-cni-cncf-io\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239413    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-var-lib-kubelet\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239448    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8bzs6\""
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-run-systemd\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-kubelet-config\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-sysctl-d\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-systemd\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-system-cni-dir\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-socket-dir-parent\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b6qk\" (UniqueName: \"kubernetes.io/projected/8a9e3f2d-f231-4b9c-be15-f6c370260974-kube-api-access-6b6qk\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-cnibin\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnvc\" (UniqueName: \"kubernetes.io/projected/c981050d-084b-424f-814b-d721448e10c9-kube-api-access-wfnvc\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-sysctl-conf\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-var-lib-kubelet\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.239882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ffvs\" (UniqueName: \"kubernetes.io/projected/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-kube-api-access-9ffvs\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-etc-openvswitch\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.239981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-conf-dir\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c18ed6a5-1b44-4261-a9cc-c5e78164115e-ovnkube-config\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-etc-kubernetes\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-kubelet\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-log-socket\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-cni-netd\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-run-netns\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-socket-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a9e3f2d-f231-4b9c-be15-f6c370260974-tmp\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-sys-fs\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-run-netns\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-var-lib-cni-multus\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.240696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-slash\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.241315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-run-ovn-kubernetes\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.241315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-run\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.241315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-sys\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.241315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-hostroot\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.241315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-dbus\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.241315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240655    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-var-lib-cni-bin\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.241315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-run-multus-certs\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.241315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.240735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkg8\" (UniqueName: \"kubernetes.io/projected/c18ed6a5-1b44-4261-a9cc-c5e78164115e-kube-api-access-tfkg8\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.242790 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.242769    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:11.242892 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.242835    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:11.245450 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.245428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.248331 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.247953    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:11.248703 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.248685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.249451 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.249428    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:11.249546 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.249496    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:44:11.249546 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.249531    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p6fsc\""
Sep 16 18:44:11.249699 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.249681    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:44:11.250433 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.250406    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:11.250637 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.250622    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"
Sep 16 18:44:11.250719 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.250653    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mjbtd\""
Sep 16 18:44:11.250719 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.250703    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:44:11.250858 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.250709    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal"]
Sep 16 18:44:11.254107 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.254070    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gqvgp\""
Sep 16 18:44:11.254253 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.254133    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:11.254538 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.254521    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:44:11.254588 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.254572    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:44:11.260703 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.260647    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:39:10 +0000 UTC" deadline="2028-05-27 07:27:32.425171792 +0000 UTC"
Sep 16 18:44:11.260703 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.260698    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14844h43m21.164477149s"
Sep 16 18:44:11.261771 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.261755    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:11.261989 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.261969    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"]
Sep 16 18:44:11.273091 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.273068    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6qv7"
Sep 16 18:44:11.281470 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.281440    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6qv7"
Sep 16 18:44:11.331493 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.331467    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:44:11.341230 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ffvs\" (UniqueName: \"kubernetes.io/projected/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-kube-api-access-9ffvs\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.341429 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-etc-openvswitch\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.341429 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.341429 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-conf-dir\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.341429 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c18ed6a5-1b44-4261-a9cc-c5e78164115e-ovnkube-config\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.341429 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.341429 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-etc-kubernetes\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.341429 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-kubelet\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.341429 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-log-socket\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-cni-netd\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/543dcafa-704f-4d94-8a68-6e6ba60429ae-system-cni-dir\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341490    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-etc-openvswitch\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-run-netns\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341550    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-conf-dir\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64f66d84-84f6-4be7-892c-7163d7be2560-konnectivity-ca\") pod \"konnectivity-agent-hs9zv\" (UID: \"64f66d84-84f6-4be7-892c-7163d7be2560\") " pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04f86fca-369a-49ce-8bd8-b734182ec3a5-host\") pod \"node-ca-x4m4g\" (UID: \"04f86fca-369a-49ce-8bd8-b734182ec3a5\") " pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/543dcafa-704f-4d94-8a68-6e6ba60429ae-cnibin\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341626    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-kubelet\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7j8q\" (UniqueName: \"kubernetes.io/projected/543dcafa-704f-4d94-8a68-6e6ba60429ae-kube-api-access-c7j8q\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-socket-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a9e3f2d-f231-4b9c-be15-f6c370260974-tmp\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341685    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-log-socket\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64f66d84-84f6-4be7-892c-7163d7be2560-agent-certs\") pod \"konnectivity-agent-hs9zv\" (UID: \"64f66d84-84f6-4be7-892c-7163d7be2560\") " pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/543dcafa-704f-4d94-8a68-6e6ba60429ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-sys-fs\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.341892 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-cni-netd\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-run-netns\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-var-lib-cni-multus\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-slash\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-run-netns\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341511    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-etc-kubernetes\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-run-ovn-kubernetes\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641e581f-3294-4bfa-852e-8f3bc14f54df-tmp-dir\") pod \"node-resolver-6rmzs\" (UID: \"641e581f-3294-4bfa-852e-8f3bc14f54df\") " pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-run\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.341903    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-sys\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-hostroot\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.341970    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret podName:eb33c8b3-cc3f-45e3-a4a6-f154e5183033 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:11.841948219 +0000 UTC m=+2.035811889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret") pod "global-pull-secret-syncer-49tdd" (UID: "eb33c8b3-cc3f-45e3-a4a6-f154e5183033") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341976    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-hostroot\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.341447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-run-ovn-kubernetes\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342028    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-run\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342074    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-sys\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.343082 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-run-netns\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342114    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-sys-fs\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342125    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-slash\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342147    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-var-lib-cni-multus\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-dbus\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342177    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-socket-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c18ed6a5-1b44-4261-a9cc-c5e78164115e-ovnkube-config\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-var-lib-cni-bin\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-run-multus-certs\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342275    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-var-lib-cni-bin\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkg8\" (UniqueName: \"kubernetes.io/projected/c18ed6a5-1b44-4261-a9cc-c5e78164115e-kube-api-access-tfkg8\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342318    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-dbus\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342331    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-run-multus-certs\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/543dcafa-704f-4d94-8a68-6e6ba60429ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-daemon-config\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e8ec2ab-0c2f-4fff-83c4-a260a24907cc-iptables-alerter-script\") pod \"iptables-alerter-x4vvb\" (UID: \"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc\") " pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-cni-dir\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.344034 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342490    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-registration-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-sysconfig\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-host\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-var-lib-openvswitch\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342626    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-registration-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-run-openvswitch\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342731    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c18ed6a5-1b44-4261-a9cc-c5e78164115e-ovnkube-script-lib\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-device-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342831    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-host\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-systemd-units\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-cni-bin\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-var-lib-openvswitch\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-systemd-units\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c18ed6a5-1b44-4261-a9cc-c5e78164115e-env-overrides\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342976    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-device-dir\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.344896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.342978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzhg8\" (UniqueName: \"kubernetes.io/projected/04f86fca-369a-49ce-8bd8-b734182ec3a5-kube-api-access-nzhg8\") pod \"node-ca-x4m4g\" (UID: \"04f86fca-369a-49ce-8bd8-b734182ec3a5\") " pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343069    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e8ec2ab-0c2f-4fff-83c4-a260a24907cc-host-slash\") pod \"iptables-alerter-x4vvb\" (UID: \"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc\") " pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343095    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-daemon-config\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-sysconfig\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-cni-dir\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-cni-bin\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343182    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-run-openvswitch\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-cni-binary-copy\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-kubernetes\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-lib-modules\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343507    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-kubernetes\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343526    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c18ed6a5-1b44-4261-a9cc-c5e78164115e-ovnkube-script-lib\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343571    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c18ed6a5-1b44-4261-a9cc-c5e78164115e-env-overrides\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-run-ovn\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-node-log\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c18ed6a5-1b44-4261-a9cc-c5e78164115e-ovn-node-metrics-cert\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-lib-modules\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.345855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343719    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-node-log\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343718    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-run-ovn\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf8tk\" (UniqueName: \"kubernetes.io/projected/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-kube-api-access-hf8tk\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343788    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-cni-binary-copy\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/543dcafa-704f-4d94-8a68-6e6ba60429ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xf2\" (UniqueName: \"kubernetes.io/projected/4e8ec2ab-0c2f-4fff-83c4-a260a24907cc-kube-api-access-t2xf2\") pod \"iptables-alerter-x4vvb\" (UID: \"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc\") " pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343960    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.343995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-modprobe-d\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-tuned\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344061    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-run-k8s-cni-cncf-io\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/641e581f-3294-4bfa-852e-8f3bc14f54df-hosts-file\") pod \"node-resolver-6rmzs\" (UID: \"641e581f-3294-4bfa-852e-8f3bc14f54df\") " pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlr4f\" (UniqueName: \"kubernetes.io/projected/641e581f-3294-4bfa-852e-8f3bc14f54df-kube-api-access-mlr4f\") pod \"node-resolver-6rmzs\" (UID: \"641e581f-3294-4bfa-852e-8f3bc14f54df\") " pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-modprobe-d\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-var-lib-kubelet\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-run-systemd\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-run-k8s-cni-cncf-io\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.346778 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344190    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-host-var-lib-kubelet\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-kubelet-config\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344256    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-kubelet-config\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344304    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-sysctl-d\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344302    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c18ed6a5-1b44-4261-a9cc-c5e78164115e-run-systemd\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-systemd\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-system-cni-dir\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-socket-dir-parent\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344402    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-systemd\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04f86fca-369a-49ce-8bd8-b734182ec3a5-serviceca\") pod \"node-ca-x4m4g\" (UID: \"04f86fca-369a-49ce-8bd8-b734182ec3a5\") " pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-system-cni-dir\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/543dcafa-704f-4d94-8a68-6e6ba60429ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-sysctl-d\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344464    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-multus-socket-dir-parent\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b6qk\" (UniqueName: \"kubernetes.io/projected/8a9e3f2d-f231-4b9c-be15-f6c370260974-kube-api-access-6b6qk\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-cnibin\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.347653 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnvc\" (UniqueName: \"kubernetes.io/projected/c981050d-084b-424f-814b-d721448e10c9-kube-api-access-wfnvc\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.348556 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344603    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-cnibin\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.348556 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-sysctl-conf\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.348556 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-var-lib-kubelet\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.348556 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344707    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c981050d-084b-424f-814b-d721448e10c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.348556 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344841    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-var-lib-kubelet\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.348556 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.344887    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-sysctl-conf\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.349703 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.349021    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:11.349703 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.349033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a9e3f2d-f231-4b9c-be15-f6c370260974-etc-tuned\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.349703 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.349051    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:11.349703 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.349067    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wx9pt for pod openshift-network-diagnostics/network-check-target-gglxk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:11.349703 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.349158    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt podName:234b9608-1129-44b3-9803-cdedb1a0d3f8 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:11.84913595 +0000 UTC m=+2.042999617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wx9pt" (UniqueName: "kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt") pod "network-check-target-gglxk" (UID: "234b9608-1129-44b3-9803-cdedb1a0d3f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:11.349703 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.349235    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c18ed6a5-1b44-4261-a9cc-c5e78164115e-ovn-node-metrics-cert\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.349703 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.349406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a9e3f2d-f231-4b9c-be15-f6c370260974-tmp\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.352525 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.352481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkg8\" (UniqueName: \"kubernetes.io/projected/c18ed6a5-1b44-4261-a9cc-c5e78164115e-kube-api-access-tfkg8\") pod \"ovnkube-node-4x5hs\" (UID: \"c18ed6a5-1b44-4261-a9cc-c5e78164115e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.352642 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.352483    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ffvs\" (UniqueName: \"kubernetes.io/projected/ccbaaf68-77fa-4845-a85d-92b44a50a8ab-kube-api-access-9ffvs\") pod \"multus-gckt2\" (UID: \"ccbaaf68-77fa-4845-a85d-92b44a50a8ab\") " pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.352765 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.352741    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b6qk\" (UniqueName: \"kubernetes.io/projected/8a9e3f2d-f231-4b9c-be15-f6c370260974-kube-api-access-6b6qk\") pod \"tuned-lx59p\" (UID: \"8a9e3f2d-f231-4b9c-be15-f6c370260974\") " pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.353019 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.353000    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnvc\" (UniqueName: \"kubernetes.io/projected/c981050d-084b-424f-814b-d721448e10c9-kube-api-access-wfnvc\") pod \"aws-ebs-csi-driver-node-z2s7q\" (UID: \"c981050d-084b-424f-814b-d721448e10c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.445863 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.445809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e8ec2ab-0c2f-4fff-83c4-a260a24907cc-iptables-alerter-script\") pod \"iptables-alerter-x4vvb\" (UID: \"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc\") " pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.445874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.445895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzhg8\" (UniqueName: \"kubernetes.io/projected/04f86fca-369a-49ce-8bd8-b734182ec3a5-kube-api-access-nzhg8\") pod \"node-ca-x4m4g\" (UID: \"04f86fca-369a-49ce-8bd8-b734182ec3a5\") " pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.445911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e8ec2ab-0c2f-4fff-83c4-a260a24907cc-host-slash\") pod \"iptables-alerter-x4vvb\" (UID: \"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc\") " pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.445937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf8tk\" (UniqueName: \"kubernetes.io/projected/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-kube-api-access-hf8tk\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.445962    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/543dcafa-704f-4d94-8a68-6e6ba60429ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.445978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xf2\" (UniqueName: \"kubernetes.io/projected/4e8ec2ab-0c2f-4fff-83c4-a260a24907cc-kube-api-access-t2xf2\") pod \"iptables-alerter-x4vvb\" (UID: \"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc\") " pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.445996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/641e581f-3294-4bfa-852e-8f3bc14f54df-hosts-file\") pod \"node-resolver-6rmzs\" (UID: \"641e581f-3294-4bfa-852e-8f3bc14f54df\") " pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlr4f\" (UniqueName: \"kubernetes.io/projected/641e581f-3294-4bfa-852e-8f3bc14f54df-kube-api-access-mlr4f\") pod \"node-resolver-6rmzs\" (UID: \"641e581f-3294-4bfa-852e-8f3bc14f54df\") " pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04f86fca-369a-49ce-8bd8-b734182ec3a5-serviceca\") pod \"node-ca-x4m4g\" (UID: \"04f86fca-369a-49ce-8bd8-b734182ec3a5\") " pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.446069 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/543dcafa-704f-4d94-8a68-6e6ba60429ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.446065    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446137    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/543dcafa-704f-4d94-8a68-6e6ba60429ae-system-cni-dir\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.446168    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:11.946146253 +0000 UTC m=+2.140009920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/543dcafa-704f-4d94-8a68-6e6ba60429ae-system-cni-dir\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64f66d84-84f6-4be7-892c-7163d7be2560-konnectivity-ca\") pod \"konnectivity-agent-hs9zv\" (UID: \"64f66d84-84f6-4be7-892c-7163d7be2560\") " pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04f86fca-369a-49ce-8bd8-b734182ec3a5-host\") pod \"node-ca-x4m4g\" (UID: \"04f86fca-369a-49ce-8bd8-b734182ec3a5\") " pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/543dcafa-704f-4d94-8a68-6e6ba60429ae-cnibin\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7j8q\" (UniqueName: \"kubernetes.io/projected/543dcafa-704f-4d94-8a68-6e6ba60429ae-kube-api-access-c7j8q\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64f66d84-84f6-4be7-892c-7163d7be2560-agent-certs\") pod \"konnectivity-agent-hs9zv\" (UID: \"64f66d84-84f6-4be7-892c-7163d7be2560\") " pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446325    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04f86fca-369a-49ce-8bd8-b734182ec3a5-host\") pod \"node-ca-x4m4g\" (UID: \"04f86fca-369a-49ce-8bd8-b734182ec3a5\") " pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/543dcafa-704f-4d94-8a68-6e6ba60429ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446371    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/543dcafa-704f-4d94-8a68-6e6ba60429ae-cnibin\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641e581f-3294-4bfa-852e-8f3bc14f54df-tmp-dir\") pod \"node-resolver-6rmzs\" (UID: \"641e581f-3294-4bfa-852e-8f3bc14f54df\") " pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/543dcafa-704f-4d94-8a68-6e6ba60429ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.446619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446544    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e8ec2ab-0c2f-4fff-83c4-a260a24907cc-iptables-alerter-script\") pod \"iptables-alerter-x4vvb\" (UID: \"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc\") " pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.447280 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446714    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/543dcafa-704f-4d94-8a68-6e6ba60429ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.447280 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641e581f-3294-4bfa-852e-8f3bc14f54df-tmp-dir\") pod \"node-resolver-6rmzs\" (UID: \"641e581f-3294-4bfa-852e-8f3bc14f54df\") " pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.447280 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.445988    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e8ec2ab-0c2f-4fff-83c4-a260a24907cc-host-slash\") pod \"iptables-alerter-x4vvb\" (UID: \"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc\") " pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.447280 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446765    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/543dcafa-704f-4d94-8a68-6e6ba60429ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.447280 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446972    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64f66d84-84f6-4be7-892c-7163d7be2560-konnectivity-ca\") pod \"konnectivity-agent-hs9zv\" (UID: \"64f66d84-84f6-4be7-892c-7163d7be2560\") " pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:11.447280 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.446962    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/641e581f-3294-4bfa-852e-8f3bc14f54df-hosts-file\") pod \"node-resolver-6rmzs\" (UID: \"641e581f-3294-4bfa-852e-8f3bc14f54df\") " pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.447280 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.447032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/543dcafa-704f-4d94-8a68-6e6ba60429ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.447520 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.447294    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/543dcafa-704f-4d94-8a68-6e6ba60429ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.447520 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.447388    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04f86fca-369a-49ce-8bd8-b734182ec3a5-serviceca\") pod \"node-ca-x4m4g\" (UID: \"04f86fca-369a-49ce-8bd8-b734182ec3a5\") " pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.449330 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.449307    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64f66d84-84f6-4be7-892c-7163d7be2560-agent-certs\") pod \"konnectivity-agent-hs9zv\" (UID: \"64f66d84-84f6-4be7-892c-7163d7be2560\") " pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:11.455141 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.455106    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xf2\" (UniqueName: \"kubernetes.io/projected/4e8ec2ab-0c2f-4fff-83c4-a260a24907cc-kube-api-access-t2xf2\") pod \"iptables-alerter-x4vvb\" (UID: \"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc\") " pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.455295 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.455176    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzhg8\" (UniqueName: \"kubernetes.io/projected/04f86fca-369a-49ce-8bd8-b734182ec3a5-kube-api-access-nzhg8\") pod \"node-ca-x4m4g\" (UID: \"04f86fca-369a-49ce-8bd8-b734182ec3a5\") " pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.455637 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.455614    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlr4f\" (UniqueName: \"kubernetes.io/projected/641e581f-3294-4bfa-852e-8f3bc14f54df-kube-api-access-mlr4f\") pod \"node-resolver-6rmzs\" (UID: \"641e581f-3294-4bfa-852e-8f3bc14f54df\") " pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.455721 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.455689    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf8tk\" (UniqueName: \"kubernetes.io/projected/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-kube-api-access-hf8tk\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:11.456120 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.456094    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7j8q\" (UniqueName: \"kubernetes.io/projected/543dcafa-704f-4d94-8a68-6e6ba60429ae-kube-api-access-c7j8q\") pod \"multus-additional-cni-plugins-m8fxx\" (UID: \"543dcafa-704f-4d94-8a68-6e6ba60429ae\") " pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.487540 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.487494    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b26e3b59ca8b326d0e3c11878453793.slice/crio-885a5c066a5bb87d3dfe66be03b9b3b9ec5e1e0747fe77335fd5bf50604eddc5 WatchSource:0}: Error finding container 885a5c066a5bb87d3dfe66be03b9b3b9ec5e1e0747fe77335fd5bf50604eddc5: Status 404 returned error can't find the container with id 885a5c066a5bb87d3dfe66be03b9b3b9ec5e1e0747fe77335fd5bf50604eddc5
Sep 16 18:44:11.488133 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.488105    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda967a84e42dca6d85c597cf3068ce6a8.slice/crio-4f7919fd318bced4e21c3c94c4414621279135983b2324449625aae8b4f4c994 WatchSource:0}: Error finding container 4f7919fd318bced4e21c3c94c4414621279135983b2324449625aae8b4f4c994: Status 404 returned error can't find the container with id 4f7919fd318bced4e21c3c94c4414621279135983b2324449625aae8b4f4c994
Sep 16 18:44:11.495247 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.495225    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:44:11.547141 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.547049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q"
Sep 16 18:44:11.560035 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.560002    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc981050d_084b_424f_814b_d721448e10c9.slice/crio-cfbfda519fba57b50001af1446f3ea2d1855255852de063f581289134d69dd77 WatchSource:0}: Error finding container cfbfda519fba57b50001af1446f3ea2d1855255852de063f581289134d69dd77: Status 404 returned error can't find the container with id cfbfda519fba57b50001af1446f3ea2d1855255852de063f581289134d69dd77
Sep 16 18:44:11.568295 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.568267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lx59p"
Sep 16 18:44:11.578287 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.578253    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9e3f2d_f231_4b9c_be15_f6c370260974.slice/crio-1da3c70a362a599b669b99b17e8eb9e728bd191a010a7d017a831084e5ed36c6 WatchSource:0}: Error finding container 1da3c70a362a599b669b99b17e8eb9e728bd191a010a7d017a831084e5ed36c6: Status 404 returned error can't find the container with id 1da3c70a362a599b669b99b17e8eb9e728bd191a010a7d017a831084e5ed36c6
Sep 16 18:44:11.593340 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.593299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gckt2"
Sep 16 18:44:11.597776 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.597755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:11.601899 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.601873    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:11.603773 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.603746    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbaaf68_77fa_4845_a85d_92b44a50a8ab.slice/crio-4d4a0646f24d0b7bc1d4501928870bd83e676846ca0f040af735d3075ecac758 WatchSource:0}: Error finding container 4d4a0646f24d0b7bc1d4501928870bd83e676846ca0f040af735d3075ecac758: Status 404 returned error can't find the container with id 4d4a0646f24d0b7bc1d4501928870bd83e676846ca0f040af735d3075ecac758
Sep 16 18:44:11.609427 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.609401    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18ed6a5_1b44_4261_a9cc_c5e78164115e.slice/crio-56abaa1d6dba6a50696fe3221e0f71ad69b4f7fc08ba790b26ba52091bfaf400 WatchSource:0}: Error finding container 56abaa1d6dba6a50696fe3221e0f71ad69b4f7fc08ba790b26ba52091bfaf400: Status 404 returned error can't find the container with id 56abaa1d6dba6a50696fe3221e0f71ad69b4f7fc08ba790b26ba52091bfaf400
Sep 16 18:44:11.613197 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.613169    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f66d84_84f6_4be7_892c_7163d7be2560.slice/crio-5743da02340ab942813063d5fff730d01f7b3c8b95112973fbdcec6d8627481e WatchSource:0}: Error finding container 5743da02340ab942813063d5fff730d01f7b3c8b95112973fbdcec6d8627481e: Status 404 returned error can't find the container with id 5743da02340ab942813063d5fff730d01f7b3c8b95112973fbdcec6d8627481e
Sep 16 18:44:11.630235 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.630199    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x4m4g"
Sep 16 18:44:11.637984 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.637960    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:11.641460 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.641426    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f86fca_369a_49ce_8bd8_b734182ec3a5.slice/crio-51acb3ed3defce95a6ba5e59ea66fa620883fb260aa4a6f4c3d1ca9609fab778 WatchSource:0}: Error finding container 51acb3ed3defce95a6ba5e59ea66fa620883fb260aa4a6f4c3d1ca9609fab778: Status 404 returned error can't find the container with id 51acb3ed3defce95a6ba5e59ea66fa620883fb260aa4a6f4c3d1ca9609fab778
Sep 16 18:44:11.647040 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.647014    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m8fxx"
Sep 16 18:44:11.656491 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.656461    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543dcafa_704f_4d94_8a68_6e6ba60429ae.slice/crio-470c623123c3142bd65ce58d27c11a633fe83559667954c53cbf462bbd840e2a WatchSource:0}: Error finding container 470c623123c3142bd65ce58d27c11a633fe83559667954c53cbf462bbd840e2a: Status 404 returned error can't find the container with id 470c623123c3142bd65ce58d27c11a633fe83559667954c53cbf462bbd840e2a
Sep 16 18:44:11.665907 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.665876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x4vvb"
Sep 16 18:44:11.674684 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.674620    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6rmzs"
Sep 16 18:44:11.677353 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.677300    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8ec2ab_0c2f_4fff_83c4_a260a24907cc.slice/crio-397b1c73090ab8c9330249bceddfd1db2865faa3fce7801f7577ec2e621888c1 WatchSource:0}: Error finding container 397b1c73090ab8c9330249bceddfd1db2865faa3fce7801f7577ec2e621888c1: Status 404 returned error can't find the container with id 397b1c73090ab8c9330249bceddfd1db2865faa3fce7801f7577ec2e621888c1
Sep 16 18:44:11.684402 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:11.684368    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641e581f_3294_4bfa_852e_8f3bc14f54df.slice/crio-77bfae9e9f8d1e30afc9648f6232002ebbda2736300a27f9869caae424315f3a WatchSource:0}: Error finding container 77bfae9e9f8d1e30afc9648f6232002ebbda2736300a27f9869caae424315f3a: Status 404 returned error can't find the container with id 77bfae9e9f8d1e30afc9648f6232002ebbda2736300a27f9869caae424315f3a
Sep 16 18:44:11.686344 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.686323    2576 kuberuntime_manager.go:1459] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-6rmzs_openshift-dns(641e581f-3294-4bfa-852e-8f3bc14f54df): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 16 18:44:11.686426 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.686358    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-6rmzs" podUID="641e581f-3294-4bfa-852e-8f3bc14f54df"
Sep 16 18:44:11.730805 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.730773    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:11.850051 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.849930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:11.850051 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.850039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:11.850277 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.850134    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:11.850277 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.850196    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret podName:eb33c8b3-cc3f-45e3-a4a6-f154e5183033 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:12.850180931 +0000 UTC m=+3.044044581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret") pod "global-pull-secret-syncer-49tdd" (UID: "eb33c8b3-cc3f-45e3-a4a6-f154e5183033") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:11.850277 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.850132    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:11.850277 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.850245    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:11.850277 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.850260    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wx9pt for pod openshift-network-diagnostics/network-check-target-gglxk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:11.850473 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.850323    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt podName:234b9608-1129-44b3-9803-cdedb1a0d3f8 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:12.850305465 +0000 UTC m=+3.044169118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx9pt" (UniqueName: "kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt") pod "network-check-target-gglxk" (UID: "234b9608-1129-44b3-9803-cdedb1a0d3f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:11.950485 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:11.950437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:11.950681 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.950655    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:11.950762 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:11.950752    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:12.950729032 +0000 UTC m=+3.144592692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:12.175647 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.175607    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:12.284941 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.284852    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:11 +0000 UTC" deadline="2028-05-12 13:56:13.516517463 +0000 UTC"
Sep 16 18:44:12.284941 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.284920    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14491h12m1.231602301s"
Sep 16 18:44:12.332379 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.329834    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:12.332379 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.330014    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:12.332379 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.330518    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:12.332379 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.330639    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:12.355672 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.355589    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x4vvb" event={"ID":"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc","Type":"ContainerStarted","Data":"397b1c73090ab8c9330249bceddfd1db2865faa3fce7801f7577ec2e621888c1"}
Sep 16 18:44:12.366550 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.366503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8fxx" event={"ID":"543dcafa-704f-4d94-8a68-6e6ba60429ae","Type":"ContainerStarted","Data":"470c623123c3142bd65ce58d27c11a633fe83559667954c53cbf462bbd840e2a"}
Sep 16 18:44:12.379998 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.379952    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerStarted","Data":"56abaa1d6dba6a50696fe3221e0f71ad69b4f7fc08ba790b26ba52091bfaf400"}
Sep 16 18:44:12.396059 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.396005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gckt2" event={"ID":"ccbaaf68-77fa-4845-a85d-92b44a50a8ab","Type":"ContainerStarted","Data":"4d4a0646f24d0b7bc1d4501928870bd83e676846ca0f040af735d3075ecac758"}
Sep 16 18:44:12.406661 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.406616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q" event={"ID":"c981050d-084b-424f-814b-d721448e10c9","Type":"ContainerStarted","Data":"cfbfda519fba57b50001af1446f3ea2d1855255852de063f581289134d69dd77"}
Sep 16 18:44:12.428663 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.428557    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" event={"ID":"4b26e3b59ca8b326d0e3c11878453793","Type":"ContainerStarted","Data":"885a5c066a5bb87d3dfe66be03b9b3b9ec5e1e0747fe77335fd5bf50604eddc5"}
Sep 16 18:44:12.442437 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.442378    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6rmzs" event={"ID":"641e581f-3294-4bfa-852e-8f3bc14f54df","Type":"ContainerStarted","Data":"77bfae9e9f8d1e30afc9648f6232002ebbda2736300a27f9869caae424315f3a"}
Sep 16 18:44:12.451603 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.451557    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0444a9580f430c7c6b52f78667a80a82198c825805f1f2004fdfdf9aed2b95f5\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-6rmzs" podUID="641e581f-3294-4bfa-852e-8f3bc14f54df"
Sep 16 18:44:12.487286 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.487137    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x4m4g" event={"ID":"04f86fca-369a-49ce-8bd8-b734182ec3a5","Type":"ContainerStarted","Data":"51acb3ed3defce95a6ba5e59ea66fa620883fb260aa4a6f4c3d1ca9609fab778"}
Sep 16 18:44:12.510232 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.510184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hs9zv" event={"ID":"64f66d84-84f6-4be7-892c-7163d7be2560","Type":"ContainerStarted","Data":"5743da02340ab942813063d5fff730d01f7b3c8b95112973fbdcec6d8627481e"}
Sep 16 18:44:12.518780 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.518738    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lx59p" event={"ID":"8a9e3f2d-f231-4b9c-be15-f6c370260974","Type":"ContainerStarted","Data":"1da3c70a362a599b669b99b17e8eb9e728bd191a010a7d017a831084e5ed36c6"}
Sep 16 18:44:12.536961 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.536918    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" event={"ID":"a967a84e42dca6d85c597cf3068ce6a8","Type":"ContainerStarted","Data":"4f7919fd318bced4e21c3c94c4414621279135983b2324449625aae8b4f4c994"}
Sep 16 18:44:12.859998 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.859524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:12.859998 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.859599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:12.859998 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.859735    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:12.859998 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.859788    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:12.859998 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.859808    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:12.859998 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.859823    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wx9pt for pod openshift-network-diagnostics/network-check-target-gglxk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:12.859998 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.859824    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret podName:eb33c8b3-cc3f-45e3-a4a6-f154e5183033 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:14.859796582 +0000 UTC m=+5.053660247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret") pod "global-pull-secret-syncer-49tdd" (UID: "eb33c8b3-cc3f-45e3-a4a6-f154e5183033") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:12.859998 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.859909    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt podName:234b9608-1129-44b3-9803-cdedb1a0d3f8 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:14.859892606 +0000 UTC m=+5.053756261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx9pt" (UniqueName: "kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt") pod "network-check-target-gglxk" (UID: "234b9608-1129-44b3-9803-cdedb1a0d3f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:12.960414 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:12.960361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:12.960615 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.960577    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:12.960681 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:12.960649    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:14.96062862 +0000 UTC m=+5.154492275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:13.285916 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:13.285836    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:11 +0000 UTC" deadline="2028-04-20 13:24:28.097166145 +0000 UTC"
Sep 16 18:44:13.285916 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:13.285909    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13962h40m14.811262446s"
Sep 16 18:44:13.329817 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:13.329537    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:13.329817 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:13.329682    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:14.331479 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:14.331447    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:14.332088 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.331587    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:14.332088 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:14.331995    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:14.332253 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.332098    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:14.876159 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:14.876113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:14.876340 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:14.876199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:14.876340 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.876318    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:14.876482 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.876378    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret podName:eb33c8b3-cc3f-45e3-a4a6-f154e5183033 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:18.876360592 +0000 UTC m=+9.070224256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret") pod "global-pull-secret-syncer-49tdd" (UID: "eb33c8b3-cc3f-45e3-a4a6-f154e5183033") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:14.876813 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.876790    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:14.876813 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.876816    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:14.876996 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.876829    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wx9pt for pod openshift-network-diagnostics/network-check-target-gglxk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:14.876996 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.876890    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt podName:234b9608-1129-44b3-9803-cdedb1a0d3f8 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:18.876873938 +0000 UTC m=+9.070737591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx9pt" (UniqueName: "kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt") pod "network-check-target-gglxk" (UID: "234b9608-1129-44b3-9803-cdedb1a0d3f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:14.977176 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:14.977131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:14.977457 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.977356    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:14.977457 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:14.977430    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:18.977410865 +0000 UTC m=+9.171274518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:15.329917 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:15.329626    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:15.329917 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:15.329765    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:16.329285 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:16.329248    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:16.329989 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:16.329390    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:16.329989 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:16.329865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:16.330104 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:16.329996    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:17.329696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:17.329649    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:17.330255 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:17.329787    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:18.329484 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:18.328921    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:18.329484 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:18.328933    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:18.329484 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:18.329129    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:18.329484 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:18.329336    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:18.911954 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:18.911912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:18.912147 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:18.911987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:18.912147 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:18.912057    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:18.912147 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:18.912120    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:18.912147 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:18.912132    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret podName:eb33c8b3-cc3f-45e3-a4a6-f154e5183033 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:26.912111548 +0000 UTC m=+17.105975212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret") pod "global-pull-secret-syncer-49tdd" (UID: "eb33c8b3-cc3f-45e3-a4a6-f154e5183033") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:18.912147 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:18.912135    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:18.912147 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:18.912149    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wx9pt for pod openshift-network-diagnostics/network-check-target-gglxk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:18.912459 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:18.912185    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt podName:234b9608-1129-44b3-9803-cdedb1a0d3f8 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:26.912174176 +0000 UTC m=+17.106037828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx9pt" (UniqueName: "kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt") pod "network-check-target-gglxk" (UID: "234b9608-1129-44b3-9803-cdedb1a0d3f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:19.012849 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:19.012796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:19.013053 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:19.012987    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:19.013122 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:19.013076    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:27.013052054 +0000 UTC m=+17.206915707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:19.329425 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:19.329384    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:19.329623 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:19.329526    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:20.330367 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:20.330325    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:20.330999 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:20.330453    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:20.330999 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:20.330464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:20.330999 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:20.330545    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:21.329193 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:21.329098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:21.329367 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:21.329227    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:22.329072 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:22.329036    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:22.329519 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:22.329048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:22.329519 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:22.329195    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:22.329519 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:22.329257    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:23.329333 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:23.329234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:23.329815 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:23.329366    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:24.329060 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:24.329019    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:24.329254 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:24.329029    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:24.329254 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:24.329162    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:24.329370 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:24.329264    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:25.328957 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:25.328918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:25.329149 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:25.329032    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:26.329543 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:26.329507    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:26.329974 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:26.329507    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:26.329974 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:26.329620    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:26.329974 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:26.329709    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:26.977220 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:26.977183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:26.977439 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:26.977247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:26.977439 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:26.977359    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:26.977439 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:26.977396    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:26.977439 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:26.977412    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:26.977439 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:26.977425    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wx9pt for pod openshift-network-diagnostics/network-check-target-gglxk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:26.977439 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:26.977433    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret podName:eb33c8b3-cc3f-45e3-a4a6-f154e5183033 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:42.977419061 +0000 UTC m=+33.171282710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret") pod "global-pull-secret-syncer-49tdd" (UID: "eb33c8b3-cc3f-45e3-a4a6-f154e5183033") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:26.977721 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:26.977465    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt podName:234b9608-1129-44b3-9803-cdedb1a0d3f8 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:42.977455152 +0000 UTC m=+33.171318814 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx9pt" (UniqueName: "kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt") pod "network-check-target-gglxk" (UID: "234b9608-1129-44b3-9803-cdedb1a0d3f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:27.078668 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:27.078575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:27.078875 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:27.078776    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:27.078954 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:27.078876    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:43.078853188 +0000 UTC m=+33.272716843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:27.329510 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:27.329418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:27.329694 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:27.329548    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:28.329355 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:28.329318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:28.329615 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:28.329469    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:28.329615 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:28.329529    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:28.329728 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:28.329669    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:29.329122 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:29.329082    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:29.329323 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:29.329196    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:30.329924 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:30.329877    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:30.330385 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:30.329984    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:30.330385 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:30.330032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:30.330385 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:30.330140    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:31.329904 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.329565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:31.330067 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:31.330004    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:31.594306 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.594266    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gckt2" event={"ID":"ccbaaf68-77fa-4845-a85d-92b44a50a8ab","Type":"ContainerStarted","Data":"d2a58ae65ae6ca01b318585c52f9eaaf3db31305a9dee89ccc2f3c7c9da40a18"}
Sep 16 18:44:31.596165 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.596132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q" event={"ID":"c981050d-084b-424f-814b-d721448e10c9","Type":"ContainerStarted","Data":"99adcb94f52e0b29c8bcff8ff179fc425e32ae0f230a0696d991fc1dbc599885"}
Sep 16 18:44:31.598007 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.597969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" event={"ID":"4b26e3b59ca8b326d0e3c11878453793","Type":"ContainerStarted","Data":"4427ae8517284164efacdaa6ea094f60880da24154a0f44fdac50a9fbedce01a"}
Sep 16 18:44:31.599822 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.599796    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6rmzs" event={"ID":"641e581f-3294-4bfa-852e-8f3bc14f54df","Type":"ContainerStarted","Data":"c016aaf726198802e318f4eedd101c7b6e13877bfe49ae414fc093bf7013095f"}
Sep 16 18:44:31.601286 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.601259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x4m4g" event={"ID":"04f86fca-369a-49ce-8bd8-b734182ec3a5","Type":"ContainerStarted","Data":"08dd943e4c96f8ca82f3e71c2450c8836162a911f55ffa89538c672c414106d1"}
Sep 16 18:44:31.603080 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.603051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hs9zv" event={"ID":"64f66d84-84f6-4be7-892c-7163d7be2560","Type":"ContainerStarted","Data":"d62f3c901448f1fd03bdd160cec478bb88a8caccf9c0112340d84c86f15e5573"}
Sep 16 18:44:31.604867 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.604826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lx59p" event={"ID":"8a9e3f2d-f231-4b9c-be15-f6c370260974","Type":"ContainerStarted","Data":"fda5db2445c2f4d6c9a5c12c58f8209607c5169ee871fb052206f68c1c8559cb"}
Sep 16 18:44:31.606773 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.606736    2576 generic.go:356] "Generic (PLEG): container finished" podID="a967a84e42dca6d85c597cf3068ce6a8" containerID="cb7c6b8df549019c80330d56f6093f0b0197e857066c4becb93c5dd237996e63" exitCode=0
Sep 16 18:44:31.606940 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.606795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" event={"ID":"a967a84e42dca6d85c597cf3068ce6a8","Type":"ContainerDied","Data":"cb7c6b8df549019c80330d56f6093f0b0197e857066c4becb93c5dd237996e63"}
Sep 16 18:44:31.608781 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.608742    2576 generic.go:356] "Generic (PLEG): container finished" podID="543dcafa-704f-4d94-8a68-6e6ba60429ae" containerID="37e82626cab7169f5a7522b5bf71eaa4eabb3ebe59a280d48e167445bbf268b3" exitCode=0
Sep 16 18:44:31.608919 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.608819    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8fxx" event={"ID":"543dcafa-704f-4d94-8a68-6e6ba60429ae","Type":"ContainerDied","Data":"37e82626cab7169f5a7522b5bf71eaa4eabb3ebe59a280d48e167445bbf268b3"}
Sep 16 18:44:31.615335 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.615308    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:44:31.615850 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.615741    2576 generic.go:356] "Generic (PLEG): container finished" podID="c18ed6a5-1b44-4261-a9cc-c5e78164115e" containerID="a9cbb167613d12aaa5ea38e69bd10672e082290780d0c4ac53a5f8254fb2ce22" exitCode=1
Sep 16 18:44:31.615850 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.615792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerStarted","Data":"2675bdd77817204a8fc35f03ed75051e8096ac60cc0452f790cb5a31fe57c5b7"}
Sep 16 18:44:31.616015 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.615894    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerStarted","Data":"b2a6be552bdd28c920d914de2f3a92a214be7849c2d2c0a3ade08aa44f315c01"}
Sep 16 18:44:31.616015 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.615949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerStarted","Data":"1d17b73553a60b2f83500e5ed7b70549bd624483bd55f2d83503c626d1cfc48f"}
Sep 16 18:44:31.616015 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.615997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerStarted","Data":"65ede234a9300633a95a5649cec5712b975019d2318acf37cb8c39aa82e66b09"}
Sep 16 18:44:31.616161 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.616043    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerDied","Data":"a9cbb167613d12aaa5ea38e69bd10672e082290780d0c4ac53a5f8254fb2ce22"}
Sep 16 18:44:31.616161 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.616096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerStarted","Data":"5ba56439a2e83081c5a58d585a8ad0fcb3f50adf6cd2d72bb5535e846fd4ab2d"}
Sep 16 18:44:31.628296 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.628209    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" podStartSLOduration=20.628185099 podStartE2EDuration="20.628185099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:31.628112771 +0000 UTC m=+21.821976443" watchObservedRunningTime="2026-09-16 18:44:31.628185099 +0000 UTC m=+21.822048771"
Sep 16 18:44:31.628858 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.628790    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gckt2" podStartSLOduration=2.466070077 podStartE2EDuration="21.628779464s" totalImagesPullingTime="19.162709387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:31.613580536 +0000 UTC m=+21.807444209" watchObservedRunningTime="2026-09-16 18:44:31.628779464 +0000 UTC m=+21.822643136"
Sep 16 18:44:31.644040 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.643971    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lx59p" podStartSLOduration=2.808560039 podStartE2EDuration="21.643948678s" totalImagesPullingTime="18.835388639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:31.643307118 +0000 UTC m=+21.837170783" watchObservedRunningTime="2026-09-16 18:44:31.643948678 +0000 UTC m=+21.837812350"
Sep 16 18:44:31.657228 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.657160    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x4m4g" podStartSLOduration=2.886150581 podStartE2EDuration="21.6571374s" totalImagesPullingTime="18.770986819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:31.657006336 +0000 UTC m=+21.850870009" watchObservedRunningTime="2026-09-16 18:44:31.6571374 +0000 UTC m=+21.851001072"
Sep 16 18:44:31.680624 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.680565    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hs9zv" podStartSLOduration=2.8812028979999997 podStartE2EDuration="21.680547052s" totalImagesPullingTime="18.799344154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:31.680501801 +0000 UTC m=+21.874365474" watchObservedRunningTime="2026-09-16 18:44:31.680547052 +0000 UTC m=+21.874410780"
Sep 16 18:44:31.722818 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:31.722707    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6rmzs" podStartSLOduration=1.69894416 podStartE2EDuration="20.722684538s" totalImagesPullingTime="19.023740378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-16 18:44:31.721795945 +0000 UTC m=+21.915659617" watchObservedRunningTime="2026-09-16 18:44:31.722684538 +0000 UTC m=+21.916548211"
Sep 16 18:44:32.329975 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:32.329938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:32.330170 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:32.330092    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:32.330863 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:32.330496    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:32.330863 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:32.330591    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:32.432073 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:32.432041    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:44:32.622327 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:32.621122    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x4vvb" event={"ID":"4e8ec2ab-0c2f-4fff-83c4-a260a24907cc","Type":"ContainerStarted","Data":"35c85a77c2de2f6949a4018d92048a2612f2d5fe3eb96d2d987fd9167d7a8b76"}
Sep 16 18:44:32.625600 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:32.625286    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q" event={"ID":"c981050d-084b-424f-814b-d721448e10c9","Type":"ContainerStarted","Data":"d310287c235f9521f4c34cb9cbf041adebec84f0bd8e5ed6bdfad3db4b747754"}
Sep 16 18:44:32.649388 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:32.649326    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x4vvb" podStartSLOduration=3.915657203 podStartE2EDuration="22.649302924s" totalImagesPullingTime="18.733645721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:32.635789832 +0000 UTC m=+22.829653505" watchObservedRunningTime="2026-09-16 18:44:32.649302924 +0000 UTC m=+22.843166596"
Sep 16 18:44:33.073477 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.073437    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:33.074170 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.074143    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:33.313577 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.313460    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:44:32.432066016Z","UUID":"41b7d2b3-26a7-4d8b-9fd4-5eca236f44ba","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:44:33.315302 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.315276    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:44:33.315450 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.315315    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:44:33.329392 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.329165    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:33.329563 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:33.329424    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:33.629904 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.629735    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" event={"ID":"a967a84e42dca6d85c597cf3068ce6a8","Type":"ContainerStarted","Data":"28b12687bf886377d864d9ce54258c131affc98c06714be9dcc81ed070080d07"}
Sep 16 18:44:33.634051 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.634021    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:44:33.634496 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.634446    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerStarted","Data":"d7f51bcf9f94b9e19e2eb2d01d373a67598fab3b9bcce87123a319932449d1b5"}
Sep 16 18:44:33.634957 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.634923    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:33.635359 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.635337    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hs9zv"
Sep 16 18:44:33.649109 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:33.649047    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" podStartSLOduration=22.649025505 podStartE2EDuration="22.649025505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:33.644772332 +0000 UTC m=+23.838636004" watchObservedRunningTime="2026-09-16 18:44:33.649025505 +0000 UTC m=+23.842889180"
Sep 16 18:44:34.329755 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:34.329716    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:34.329954 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:34.329769    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:34.329954 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:34.329902    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:34.330039 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:34.330006    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:34.639664 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:34.639549    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q" event={"ID":"c981050d-084b-424f-814b-d721448e10c9","Type":"ContainerStarted","Data":"523843bea7d581cdd4394adb75856ba5a87e3df7ae494b9500d548d559bae2f0"}
Sep 16 18:44:34.656077 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:34.656010    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2s7q" podStartSLOduration=3.443649579 podStartE2EDuration="24.655992063s" totalImagesPullingTime="21.212342484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:34.655975877 +0000 UTC m=+24.849839559" watchObservedRunningTime="2026-09-16 18:44:34.655992063 +0000 UTC m=+24.849855729"
Sep 16 18:44:35.329405 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:35.329371    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:35.329606 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:35.329494    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:36.328981 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.328944    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:36.329417 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.328944    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:36.329417 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:36.329062    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:36.329417 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:36.329130    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:36.646351 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.646094    2576 generic.go:356] "Generic (PLEG): container finished" podID="543dcafa-704f-4d94-8a68-6e6ba60429ae" containerID="ba8806bd947c82600140ff981d8dbd33e02bf2fcb2c76be374e067f88c09d16d" exitCode=0
Sep 16 18:44:36.646351 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.646179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8fxx" event={"ID":"543dcafa-704f-4d94-8a68-6e6ba60429ae","Type":"ContainerDied","Data":"ba8806bd947c82600140ff981d8dbd33e02bf2fcb2c76be374e067f88c09d16d"}
Sep 16 18:44:36.650355 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.650339    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:44:36.650757 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.650733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerStarted","Data":"739149a59d4892e6bc41340eb37f9c9fd3f09bb9a637a9898d3a7057d35979d3"}
Sep 16 18:44:36.651052 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.651012    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:36.651147 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.651091    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:36.651147 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.651141    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:36.651343 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.651219    2576 scope.go:117] "RemoveContainer" containerID="a9cbb167613d12aaa5ea38e69bd10672e082290780d0c4ac53a5f8254fb2ce22"
Sep 16 18:44:36.668603 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.668577    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:36.672334 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:36.672310    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:44:37.329753 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:37.329709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:37.330541 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:37.329875    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:37.655918 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:37.655880    2576 generic.go:356] "Generic (PLEG): container finished" podID="543dcafa-704f-4d94-8a68-6e6ba60429ae" containerID="37c0bb363f8cb615afd0886e128f925ec53af07f560da0b695d657fae0d0c422" exitCode=0
Sep 16 18:44:37.656122 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:37.655960    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8fxx" event={"ID":"543dcafa-704f-4d94-8a68-6e6ba60429ae","Type":"ContainerDied","Data":"37c0bb363f8cb615afd0886e128f925ec53af07f560da0b695d657fae0d0c422"}
Sep 16 18:44:37.660245 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:37.660225    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:44:37.660654 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:37.660630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" event={"ID":"c18ed6a5-1b44-4261-a9cc-c5e78164115e","Type":"ContainerStarted","Data":"a77ba92e62fa1330f6fb5228987d7919e65e3f5c2c7fb12e3ed72f2d87ace714"}
Sep 16 18:44:37.700415 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:37.700330    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs" podStartSLOduration=8.863499855 podStartE2EDuration="27.70030867s" totalImagesPullingTime="18.836808815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:37.699085989 +0000 UTC m=+27.892949658" watchObservedRunningTime="2026-09-16 18:44:37.70030867 +0000 UTC m=+27.894172342"
Sep 16 18:44:38.094477 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:38.094441    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-49tdd"]
Sep 16 18:44:38.094663 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:38.094617    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:38.094737 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:38.094717    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:38.097454 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:38.097420    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gglxk"]
Sep 16 18:44:38.097627 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:38.097544    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:38.097694 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:38.097650    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:38.098068 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:38.098047    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fzv4f"]
Sep 16 18:44:38.098179 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:38.098165    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:38.098292 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:38.098269    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:38.664993 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:38.664960    2576 generic.go:356] "Generic (PLEG): container finished" podID="543dcafa-704f-4d94-8a68-6e6ba60429ae" containerID="0fef582a176cfd1866ed41e44603fecfe186bee0c5a616d7913eaa8a85b331a6" exitCode=0
Sep 16 18:44:38.665607 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:38.665032    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8fxx" event={"ID":"543dcafa-704f-4d94-8a68-6e6ba60429ae","Type":"ContainerDied","Data":"0fef582a176cfd1866ed41e44603fecfe186bee0c5a616d7913eaa8a85b331a6"}
Sep 16 18:44:39.329373 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:39.329341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:39.329373 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:39.329357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:39.329591 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:39.329468    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:39.329660 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:39.329603    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:40.331042 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:40.330366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:40.331042 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:40.330708    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:41.329079 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:41.329036    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:41.329236 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:41.329191    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:41.329282 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:41.329228    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:41.329350 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:41.329335    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:42.329170 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:42.329134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:42.329667 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:42.329272    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:43.000598 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:43.000572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:43.000676 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:43.000647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:43.000727 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.000710    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:43.000727 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.000727    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:43.000819 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.000736    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wx9pt for pod openshift-network-diagnostics/network-check-target-gglxk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:43.000819 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.000774    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:43.000929 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.000784    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt podName:234b9608-1129-44b3-9803-cdedb1a0d3f8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.000770321 +0000 UTC m=+65.194634003 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx9pt" (UniqueName: "kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt") pod "network-check-target-gglxk" (UID: "234b9608-1129-44b3-9803-cdedb1a0d3f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:43.000929 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.000866    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret podName:eb33c8b3-cc3f-45e3-a4a6-f154e5183033 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.000833876 +0000 UTC m=+65.194697547 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret") pod "global-pull-secret-syncer-49tdd" (UID: "eb33c8b3-cc3f-45e3-a4a6-f154e5183033") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:43.101453 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:43.101418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:43.101611 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.101541    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:43.101611 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.101596    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.101582245 +0000 UTC m=+65.295445907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:43.329594 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:43.329497    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:43.329594 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:43.329497    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:43.330244 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.329651    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:44:43.330244 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:43.329669    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tdd" podUID="eb33c8b3-cc3f-45e3-a4a6-f154e5183033"
Sep 16 18:44:43.681202 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:43.681166    2576 generic.go:356] "Generic (PLEG): container finished" podID="543dcafa-704f-4d94-8a68-6e6ba60429ae" containerID="145db092613ac4e588cd1f24ce8abc67cdfeb46fe17eb99a71db6f3280bcd102" exitCode=0
Sep 16 18:44:43.681413 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:43.681227    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8fxx" event={"ID":"543dcafa-704f-4d94-8a68-6e6ba60429ae","Type":"ContainerDied","Data":"145db092613ac4e588cd1f24ce8abc67cdfeb46fe17eb99a71db6f3280bcd102"}
Sep 16 18:44:44.329427 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.329382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:44.329551 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:44.329514    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gglxk" podUID="234b9608-1129-44b3-9803-cdedb1a0d3f8"
Sep 16 18:44:44.604331 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.604257    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeReady"
Sep 16 18:44:44.604710 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.604405    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:44:44.638980 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.638941    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7787c9b66b-2tlsh"]
Sep 16 18:44:44.655263 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.655224    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"]
Sep 16 18:44:44.655459 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.655353    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.658115 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.658047    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:44:44.658115 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.658101    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:44:44.658772 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.658499    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lpjvx\""
Sep 16 18:44:44.658772 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.658534    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:44:44.663998 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.663970    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:44:44.669725 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.669697    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-422rl"]
Sep 16 18:44:44.669885 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.669820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:44.672295 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.672269    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:44:44.672295 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.672285    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:44:44.672473 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.672307    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t7mvb\""
Sep 16 18:44:44.687971 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.687942    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"]
Sep 16 18:44:44.688162 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.687983    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-422rl"]
Sep 16 18:44:44.688162 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.688003    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7787c9b66b-2tlsh"]
Sep 16 18:44:44.688162 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.688026    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tz7mv"]
Sep 16 18:44:44.688162 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.688061    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-422rl"
Sep 16 18:44:44.689071 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.689044    2576 generic.go:356] "Generic (PLEG): container finished" podID="543dcafa-704f-4d94-8a68-6e6ba60429ae" containerID="b8323f7d1f0e48df07e53c5f6d9d0eb9fc88ce9e4923590648124a0adb5f5af0" exitCode=0
Sep 16 18:44:44.690520 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.690497    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ftmw5\""
Sep 16 18:44:44.690616 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.690495    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:44:44.690616 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.690554    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:44:44.706449 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.706401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8fxx" event={"ID":"543dcafa-704f-4d94-8a68-6e6ba60429ae","Type":"ContainerDied","Data":"b8323f7d1f0e48df07e53c5f6d9d0eb9fc88ce9e4923590648124a0adb5f5af0"}
Sep 16 18:44:44.706605 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.706456    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tz7mv"
Sep 16 18:44:44.706605 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.706503    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kv4z5"]
Sep 16 18:44:44.708825 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.708806    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:44:44.708990 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.708806    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:44:44.708990 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.708808    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p2cz7\""
Sep 16 18:44:44.709107 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.709017    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:44:44.732702 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.732658    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tz7mv"]
Sep 16 18:44:44.732702 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.732697    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kv4z5"]
Sep 16 18:44:44.732984 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.732807    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.735118 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.734879    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:44:44.735118 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.735008    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l7s4t\""
Sep 16 18:44:44.736186 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.736161    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:44:44.813032 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e68a197f-86c7-45e7-a856-4b5574581420-cert\") pod \"ingress-canary-tz7mv\" (UID: \"e68a197f-86c7-45e7-a856-4b5574581420\") " pod="openshift-ingress-canary/ingress-canary-tz7mv"
Sep 16 18:44:44.813169 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.813169 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-trusted-ca\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.813169 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-config-volume\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.813312 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.813312 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-image-registry-private-configuration\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.813380 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c727208-7c86-4cb0-ae4d-f43027417da6-ca-trust-extracted\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.813419 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e101c2f1-e28b-4910-8d86-9d3265060b4c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:44.813467 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:44.813501 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813475    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxcqp\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-kube-api-access-bxcqp\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.813535 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-tmp-dir\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.813535 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qcl\" (UniqueName: \"kubernetes.io/projected/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-kube-api-access-h5qcl\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.813622 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-installation-pull-secrets\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.813622 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7lz2\" (UniqueName: \"kubernetes.io/projected/243e8f33-96bd-4cbc-adae-88da4d776423-kube-api-access-j7lz2\") pod \"network-check-source-6f8b768455-422rl\" (UID: \"243e8f33-96bd-4cbc-adae-88da4d776423\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-422rl"
Sep 16 18:44:44.813695 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-bound-sa-token\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.813739 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813707    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqxs\" (UniqueName: \"kubernetes.io/projected/e68a197f-86c7-45e7-a856-4b5574581420-kube-api-access-hjqxs\") pod \"ingress-canary-tz7mv\" (UID: \"e68a197f-86c7-45e7-a856-4b5574581420\") " pod="openshift-ingress-canary/ingress-canary-tz7mv"
Sep 16 18:44:44.813739 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.813734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-certificates\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.822154 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.822120    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hs9zv_64f66d84-84f6-4be7-892c-7163d7be2560/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:44.907497 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.907471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hs9zv_64f66d84-84f6-4be7-892c-7163d7be2560/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:44.914441 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.914585 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-trusted-ca\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.914585 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-config-volume\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.914585 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.914585 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-image-registry-private-configuration\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.914585 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c727208-7c86-4cb0-ae4d-f43027417da6-ca-trust-extracted\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.914585 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:44.914577    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e101c2f1-e28b-4910-8d86-9d3265060b4c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:44.914666    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:44:45.41464613 +0000 UTC m=+35.608509803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:44.914692    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:44.914716    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxcqp\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-kube-api-access-bxcqp\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:44.914771    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:45.414752786 +0000 UTC m=+35.608616437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-tmp-dir\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qcl\" (UniqueName: \"kubernetes.io/projected/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-kube-api-access-h5qcl\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-installation-pull-secrets\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.914932 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7lz2\" (UniqueName: \"kubernetes.io/projected/243e8f33-96bd-4cbc-adae-88da4d776423-kube-api-access-j7lz2\") pod \"network-check-source-6f8b768455-422rl\" (UID: \"243e8f33-96bd-4cbc-adae-88da4d776423\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-422rl"
Sep 16 18:44:44.915451 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-bound-sa-token\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.915451 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.914984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqxs\" (UniqueName: \"kubernetes.io/projected/e68a197f-86c7-45e7-a856-4b5574581420-kube-api-access-hjqxs\") pod \"ingress-canary-tz7mv\" (UID: \"e68a197f-86c7-45e7-a856-4b5574581420\") " pod="openshift-ingress-canary/ingress-canary-tz7mv"
Sep 16 18:44:44.915451 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.915018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-certificates\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.915451 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.915050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e68a197f-86c7-45e7-a856-4b5574581420-cert\") pod \"ingress-canary-tz7mv\" (UID: \"e68a197f-86c7-45e7-a856-4b5574581420\") " pod="openshift-ingress-canary/ingress-canary-tz7mv"
Sep 16 18:44:44.915451 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.915196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-config-volume\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.915451 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.915197    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c727208-7c86-4cb0-ae4d-f43027417da6-ca-trust-extracted\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.915451 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:44.915393    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:44:44.915451 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.915445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-tmp-dir\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.915833 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:44.915456    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:45.415439166 +0000 UTC m=+35.609302839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:44:44.915833 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.915683    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e101c2f1-e28b-4910-8d86-9d3265060b4c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:44.916128 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.916107    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-trusted-ca\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.916219 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.916200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-certificates\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.919435 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.919407    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-installation-pull-secrets\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.919537 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.919408    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-image-registry-private-configuration\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.919537 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.919506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e68a197f-86c7-45e7-a856-4b5574581420-cert\") pod \"ingress-canary-tz7mv\" (UID: \"e68a197f-86c7-45e7-a856-4b5574581420\") " pod="openshift-ingress-canary/ingress-canary-tz7mv"
Sep 16 18:44:44.924955 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.924872    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-bound-sa-token\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.924955 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.924927    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqxs\" (UniqueName: \"kubernetes.io/projected/e68a197f-86c7-45e7-a856-4b5574581420-kube-api-access-hjqxs\") pod \"ingress-canary-tz7mv\" (UID: \"e68a197f-86c7-45e7-a856-4b5574581420\") " pod="openshift-ingress-canary/ingress-canary-tz7mv"
Sep 16 18:44:44.925204 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.925061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qcl\" (UniqueName: \"kubernetes.io/projected/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-kube-api-access-h5qcl\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:44.925328 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.925303    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7lz2\" (UniqueName: \"kubernetes.io/projected/243e8f33-96bd-4cbc-adae-88da4d776423-kube-api-access-j7lz2\") pod \"network-check-source-6f8b768455-422rl\" (UID: \"243e8f33-96bd-4cbc-adae-88da4d776423\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-422rl"
Sep 16 18:44:44.925490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.925471    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxcqp\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-kube-api-access-bxcqp\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:44.998046 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:44.998004    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-422rl"
Sep 16 18:44:45.016273 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.016236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tz7mv"
Sep 16 18:44:45.329478 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.329232    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:44:45.329634 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.329298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:44:45.332140 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.332090    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:45.332140 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.332134    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7prs7\""
Sep 16 18:44:45.332140 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.332146    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:45.337092 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.337067    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-422rl"]
Sep 16 18:44:45.338293 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.338080    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tz7mv"]
Sep 16 18:44:45.340870 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:45.340831    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68a197f_86c7_45e7_a856_4b5574581420.slice/crio-478e376a6f69c34ca1372634fbf1f6be43a696488be2a0e84c15b0b7a7c60d99 WatchSource:0}: Error finding container 478e376a6f69c34ca1372634fbf1f6be43a696488be2a0e84c15b0b7a7c60d99: Status 404 returned error can't find the container with id 478e376a6f69c34ca1372634fbf1f6be43a696488be2a0e84c15b0b7a7c60d99
Sep 16 18:44:45.341891 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:44:45.341873    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243e8f33_96bd_4cbc_adae_88da4d776423.slice/crio-87da38e604bd051d76c57f8c70c24c42d3e206ba3e8d3c2f56783b07b011d88c WatchSource:0}: Error finding container 87da38e604bd051d76c57f8c70c24c42d3e206ba3e8d3c2f56783b07b011d88c: Status 404 returned error can't find the container with id 87da38e604bd051d76c57f8c70c24c42d3e206ba3e8d3c2f56783b07b011d88c
Sep 16 18:44:45.420079 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.420034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:45.420284 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.420128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:45.420284 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.420162    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:45.420284 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:45.420184    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:44:45.420284 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:45.420246    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:46.420230128 +0000 UTC m=+36.614093778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:44:45.420284 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:45.420265    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:45.420562 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:45.420315    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:44:46.42030139 +0000 UTC m=+36.614165042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:45.420562 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:45.420329    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:45.420562 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:45.420348    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:44:45.420562 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:45.420394    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:46.420383914 +0000 UTC m=+36.614247568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:44:45.697784 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.697747    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8fxx" event={"ID":"543dcafa-704f-4d94-8a68-6e6ba60429ae","Type":"ContainerStarted","Data":"95e283bb2dccd35483bc09025c47ddd38e72a5f05bd001532a291369a30403c2"}
Sep 16 18:44:45.698974 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.698949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-422rl" event={"ID":"243e8f33-96bd-4cbc-adae-88da4d776423","Type":"ContainerStarted","Data":"87da38e604bd051d76c57f8c70c24c42d3e206ba3e8d3c2f56783b07b011d88c"}
Sep 16 18:44:45.700020 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.699998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tz7mv" event={"ID":"e68a197f-86c7-45e7-a856-4b5574581420","Type":"ContainerStarted","Data":"478e376a6f69c34ca1372634fbf1f6be43a696488be2a0e84c15b0b7a7c60d99"}
Sep 16 18:44:45.718772 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.718718    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m8fxx" podStartSLOduration=6.472004772 podStartE2EDuration="35.718702331s" totalImagesPullingTime="29.246697559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:45.71815732 +0000 UTC m=+35.912020992" watchObservedRunningTime="2026-09-16 18:44:45.718702331 +0000 UTC m=+35.912566004"
Sep 16 18:44:45.845342 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.845303    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hs9zv_64f66d84-84f6-4be7-892c-7163d7be2560/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.945586 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:45.945553    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hs9zv_64f66d84-84f6-4be7-892c-7163d7be2560/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:46.329571 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:46.329540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:44:46.332589 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:46.332562    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5l9rg\""
Sep 16 18:44:46.429927 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:46.429876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:46.430137 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:46.429955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:46.430137 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:46.430089    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:46.430137 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:46.430110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:46.430256 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:46.430170    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:44:48.430148719 +0000 UTC m=+38.624012389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:46.430335 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:46.430272    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:44:46.430456 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:46.430370    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:48.43035296 +0000 UTC m=+38.624216623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:44:46.430456 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:46.430446    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:46.430553 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:46.430458    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:44:46.430553 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:46.430491    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:48.430480805 +0000 UTC m=+38.624344459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:44:48.451107 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:48.450990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:48.451107 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:48.451090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:48.451734 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:48.451122    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:48.451734 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:48.451149    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:44:48.451734 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:48.451217    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:52.45119755 +0000 UTC m=+42.645061216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:44:48.451734 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:48.451245    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:48.451734 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:48.451257    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:44:48.451734 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:48.451275    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:48.451734 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:48.451301    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:52.451289006 +0000 UTC m=+42.645152656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:44:48.451734 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:48.451338    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:44:52.451317894 +0000 UTC m=+42.645181559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:48.981050 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:48.981007    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hs9zv_64f66d84-84f6-4be7-892c-7163d7be2560/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:49.037853 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:49.037813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hs9zv_64f66d84-84f6-4be7-892c-7163d7be2560/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:49.717976 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:49.717926    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-422rl" event={"ID":"243e8f33-96bd-4cbc-adae-88da4d776423","Type":"ContainerStarted","Data":"78dc1f455782aecfad476a111c821033d5c6697a54bf5640771ccbdd8eadf400"}
Sep 16 18:44:49.719673 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:49.719639    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tz7mv" event={"ID":"e68a197f-86c7-45e7-a856-4b5574581420","Type":"ContainerStarted","Data":"52b5771f3ec4c56e97c57d321720c615bc846d5f33fdf03670df8a7eb47b57c7"}
Sep 16 18:44:49.746305 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:49.746218    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-422rl" podStartSLOduration=35.026398401 podStartE2EDuration="38.746196794s" totalImagesPullingTime="3.719798393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:49.732489637 +0000 UTC m=+39.926353310" watchObservedRunningTime="2026-09-16 18:44:49.746196794 +0000 UTC m=+39.940060473"
Sep 16 18:44:49.746709 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:49.746672    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tz7mv" podStartSLOduration=2.035721578 podStartE2EDuration="5.746666864s" totalImagesPullingTime="3.710945286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:49.745886896 +0000 UTC m=+39.939750569" watchObservedRunningTime="2026-09-16 18:44:49.746666864 +0000 UTC m=+39.940530593"
Sep 16 18:44:52.481733 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:52.481688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:44:52.481733 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:52.481738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:44:52.482314 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:52.481856    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:52.482314 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:52.481868    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:44:52.482314 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:52.481873    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:52.482314 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:52.481919    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.481904891 +0000 UTC m=+50.675768573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:44:52.482314 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:52.481931    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.481925946 +0000 UTC m=+50.675789595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:52.482314 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:44:52.481912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:44:52.482314 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:52.481975    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:44:52.482314 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:44:52.482031    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.482016211 +0000 UTC m=+50.675879864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:00.544183 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:00.544133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:45:00.544756 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:00.544201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:45:00.544756 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:00.544224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:45:00.544756 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:00.544301    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:00.544756 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:00.544313    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:00.544756 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:00.544325    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:45:00.544756 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:00.544342    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:00.544756 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:00.544372    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.544359998 +0000 UTC m=+66.738223662 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:45:00.544756 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:00.544402    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.544377529 +0000 UTC m=+66.738241194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:00.544756 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:00.544421    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.544412232 +0000 UTC m=+66.738275883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:08.676022 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:08.675991    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4x5hs"
Sep 16 18:45:15.045768 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.045705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:45:15.046213 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.045804    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:45:15.049037 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.049008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx9pt\" (UniqueName: \"kubernetes.io/projected/234b9608-1129-44b3-9803-cdedb1a0d3f8-kube-api-access-wx9pt\") pod \"network-check-target-gglxk\" (UID: \"234b9608-1129-44b3-9803-cdedb1a0d3f8\") " pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:45:15.050016 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.049990    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:15.059315 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.059283    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb33c8b3-cc3f-45e3-a4a6-f154e5183033-original-pull-secret\") pod \"global-pull-secret-syncer-49tdd\" (UID: \"eb33c8b3-cc3f-45e3-a4a6-f154e5183033\") " pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:45:15.144745 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.144711    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5l9rg\""
Sep 16 18:45:15.146511 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.146484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:45:15.148210 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.148187    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:15.152776 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.152753    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:45:15.157420 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:15.157362    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:45:15.157672 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:15.157521    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.157479259 +0000 UTC m=+129.351342923 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : secret "metrics-daemon-secret" not found
Sep 16 18:45:15.353278 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.353180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tdd"
Sep 16 18:45:15.422560 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.422527    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gglxk"]
Sep 16 18:45:15.426047 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:45:15.426014    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234b9608_1129_44b3_9803_cdedb1a0d3f8.slice/crio-aac252d8428f2b62763447d8bfbb8889c5911a2fecbb8c8f7398126441774900 WatchSource:0}: Error finding container aac252d8428f2b62763447d8bfbb8889c5911a2fecbb8c8f7398126441774900: Status 404 returned error can't find the container with id aac252d8428f2b62763447d8bfbb8889c5911a2fecbb8c8f7398126441774900
Sep 16 18:45:15.622288 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.622195    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-49tdd"]
Sep 16 18:45:15.627821 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:45:15.626181    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb33c8b3_cc3f_45e3_a4a6_f154e5183033.slice/crio-4a87039bbca3bfd7b2e4f0062500c1e5feaecfff352d8b738d4ecb798fa6bb81 WatchSource:0}: Error finding container 4a87039bbca3bfd7b2e4f0062500c1e5feaecfff352d8b738d4ecb798fa6bb81: Status 404 returned error can't find the container with id 4a87039bbca3bfd7b2e4f0062500c1e5feaecfff352d8b738d4ecb798fa6bb81
Sep 16 18:45:15.787159 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.787116    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-49tdd" event={"ID":"eb33c8b3-cc3f-45e3-a4a6-f154e5183033","Type":"ContainerStarted","Data":"4a87039bbca3bfd7b2e4f0062500c1e5feaecfff352d8b738d4ecb798fa6bb81"}
Sep 16 18:45:15.788586 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.788550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gglxk" event={"ID":"234b9608-1129-44b3-9803-cdedb1a0d3f8","Type":"ContainerStarted","Data":"accacdcf05e8994bbf7f64ffe228d93c233fee94a8834d0a5338f94b6d4c75e3"}
Sep 16 18:45:15.788739 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.788598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gglxk" event={"ID":"234b9608-1129-44b3-9803-cdedb1a0d3f8","Type":"ContainerStarted","Data":"aac252d8428f2b62763447d8bfbb8889c5911a2fecbb8c8f7398126441774900"}
Sep 16 18:45:15.788739 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.788670    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:45:15.807085 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:15.807033    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gglxk" podStartSLOduration=65.807012833 podStartE2EDuration="1m5.807012833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:15.80666518 +0000 UTC m=+66.000528846" watchObservedRunningTime="2026-09-16 18:45:15.807012833 +0000 UTC m=+66.000876506"
Sep 16 18:45:16.559250 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:16.559213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:45:16.559250 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:16.559258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:45:16.559824 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:16.559314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:45:16.559824 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:16.559380    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:16.559824 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:16.559423    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:16.559824 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:16.559442    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.559426763 +0000 UTC m=+98.753290413 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:16.559824 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:16.559464    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.559452348 +0000 UTC m=+98.753316011 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:16.559824 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:16.559515    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:16.559824 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:16.559527    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:45:16.559824 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:16.559562    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.559551949 +0000 UTC m=+98.753415616 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:45:19.801626 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:19.801532    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-49tdd" event={"ID":"eb33c8b3-cc3f-45e3-a4a6-f154e5183033","Type":"ContainerStarted","Data":"bca30c7873dc469bfdda98fbb3800046a78c0c8fc8aa41d85656a9424a9b967f"}
Sep 16 18:45:19.815638 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:19.815567    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-49tdd" podStartSLOduration=65.979052264 podStartE2EDuration="1m9.815547528s" totalImagesPullingTime="3.836495264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:19.81535292 +0000 UTC m=+70.009216596" watchObservedRunningTime="2026-09-16 18:45:19.815547528 +0000 UTC m=+70.009411201"
Sep 16 18:45:46.794909 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:46.794751    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gglxk"
Sep 16 18:45:48.607050 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:48.607008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:45:48.607050 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:48.607049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:45:48.607521 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:45:48.607101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:45:48.607521 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:48.607176    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:48.607521 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:48.607217    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:48.607521 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:48.607227    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:48.607521 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:48.607244    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:45:48.607521 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:48.607256    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:52.607238306 +0000 UTC m=+162.801101970 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:48.607521 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:48.607283    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:52.607266087 +0000 UTC m=+162.801129751 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:48.607521 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:45:48.607303    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:52.607291486 +0000 UTC m=+162.801155141 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:46:19.247351 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:46:19.247306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:46:19.247886 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:19.247426    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:46:19.247886 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:19.247488    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:21.247471075 +0000 UTC m=+251.441334728 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : secret "metrics-daemon-secret" not found
Sep 16 18:46:47.667173 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:47.667128    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh" podUID="7c727208-7c86-4cb0-ae4d-f43027417da6"
Sep 16 18:46:47.679008 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:47.678970    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb" podUID="e101c2f1-e28b-4910-8d86-9d3265060b4c"
Sep 16 18:46:47.745633 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:47.745585    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kv4z5" podUID="cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc"
Sep 16 18:46:48.021295 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:46:48.021262    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:46:48.021493 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:46:48.021262    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:46:48.341907 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:48.341806    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:46:52.687968 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:46:52.687919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:46:52.688520 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:46:52.687980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:46:52.688520 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:52.688071    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:52.688520 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:52.688082    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:52.688520 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:52.688136    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:48:54.688116845 +0000 UTC m=+284.881980508 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:52.688520 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:46:52.688158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:46:52.688520 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:52.688172    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:48:54.688151454 +0000 UTC m=+284.882015106 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:52.688520 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:52.688223    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:52.688520 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:52.688235    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:46:52.688520 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:46:52.688275    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:54.688264052 +0000 UTC m=+284.882127719 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:46:59.329297 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:46:59.329236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:47:00.329991 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:47:00.329957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:48:21.335565 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:21.335514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:48:21.336123 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:21.335669    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:48:21.336123 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:21.335743    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs podName:5d7a2ef5-a033-4dd2-89e3-bc86658a3373 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:23.335726247 +0000 UTC m=+373.529589900 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs") pod "network-metrics-daemon-fzv4f" (UID: "5d7a2ef5-a033-4dd2-89e3-bc86658a3373") : secret "metrics-daemon-secret" not found
Sep 16 18:48:40.495393 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.495310    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv"]
Sep 16 18:48:40.498135 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.498120    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv"
Sep 16 18:48:40.502010 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.501987    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:48:40.502145 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.502009    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ljfzg\""
Sep 16 18:48:40.502647 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.502627    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:48:40.503239 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.503219    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"]
Sep 16 18:48:40.506119 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.506099    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"]
Sep 16 18:48:40.506233 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.506215    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:40.508579 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.508554    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rqmtp\""
Sep 16 18:48:40.508690 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.508578    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:48:40.508832 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.508813    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:48:40.509150 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.509133    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-k6nld"]
Sep 16 18:48:40.509246 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.509234    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:48:40.509307 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.509266    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:40.512003 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.511980    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv"]
Sep 16 18:48:40.512101 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.512076    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.513163 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.513143    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:48:40.513458 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.513437    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8wftk\""
Sep 16 18:48:40.514231 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.514212    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:48:40.514354 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.514325    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:48:40.514452 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.514433    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:48:40.514996 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.514980    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:48:40.515090 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.515070    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:48:40.515090 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.515089    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:48:40.515290 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.515008    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:48:40.516464 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.516446    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gxzfr\""
Sep 16 18:48:40.519000 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.518980    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"]
Sep 16 18:48:40.520029 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.520010    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:48:40.523695 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.523673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"]
Sep 16 18:48:40.526234 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.526213    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-k6nld"]
Sep 16 18:48:40.584223 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.584182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx44p\" (UniqueName: \"kubernetes.io/projected/fb408ab8-669f-4ae8-9e41-39ddddcfcb69-kube-api-access-lx44p\") pod \"volume-data-source-validator-86695b7cc7-7tcbv\" (UID: \"fb408ab8-669f-4ae8-9e41-39ddddcfcb69\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv"
Sep 16 18:48:40.584223 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.584224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxb2h\" (UniqueName: \"kubernetes.io/projected/fec3d7e3-8425-4bc7-a513-f2a545a7512f-kube-api-access-pxb2h\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:40.584453 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.584273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:40.685308 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c017d51-ac33-45e1-83a3-0c08e71053cb-tmp\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.685308 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c017d51-ac33-45e1-83a3-0c08e71053cb-serving-cert\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.685503 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2c017d51-ac33-45e1-83a3-0c08e71053cb-snapshots\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.685503 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpql2\" (UniqueName: \"kubernetes.io/projected/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-kube-api-access-lpql2\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:40.685503 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c017d51-ac33-45e1-83a3-0c08e71053cb-trusted-ca-bundle\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.685503 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6v4\" (UniqueName: \"kubernetes.io/projected/2c017d51-ac33-45e1-83a3-0c08e71053cb-kube-api-access-sb6v4\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.685503 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx44p\" (UniqueName: \"kubernetes.io/projected/fb408ab8-669f-4ae8-9e41-39ddddcfcb69-kube-api-access-lx44p\") pod \"volume-data-source-validator-86695b7cc7-7tcbv\" (UID: \"fb408ab8-669f-4ae8-9e41-39ddddcfcb69\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv"
Sep 16 18:48:40.685661 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c017d51-ac33-45e1-83a3-0c08e71053cb-service-ca-bundle\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.685661 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxb2h\" (UniqueName: \"kubernetes.io/projected/fec3d7e3-8425-4bc7-a513-f2a545a7512f-kube-api-access-pxb2h\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:40.685661 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:40.685661 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:40.685661 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.685638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:40.685880 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:40.685776    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:48:40.685880 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:40.685858    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls podName:fec3d7e3-8425-4bc7-a513-f2a545a7512f nodeName:}" failed. No retries permitted until 2026-09-16 18:48:41.185824181 +0000 UTC m=+271.379687831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xdpkl" (UID: "fec3d7e3-8425-4bc7-a513-f2a545a7512f") : secret "samples-operator-tls" not found
Sep 16 18:48:40.696468 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.696441    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxb2h\" (UniqueName: \"kubernetes.io/projected/fec3d7e3-8425-4bc7-a513-f2a545a7512f-kube-api-access-pxb2h\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:40.699062 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.699036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx44p\" (UniqueName: \"kubernetes.io/projected/fb408ab8-669f-4ae8-9e41-39ddddcfcb69-kube-api-access-lx44p\") pod \"volume-data-source-validator-86695b7cc7-7tcbv\" (UID: \"fb408ab8-669f-4ae8-9e41-39ddddcfcb69\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv"
Sep 16 18:48:40.786540 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.786447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c017d51-ac33-45e1-83a3-0c08e71053cb-service-ca-bundle\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.786540 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.786518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:40.786722 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.786562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:40.786722 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:40.786677    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:40.786787 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.786719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c017d51-ac33-45e1-83a3-0c08e71053cb-tmp\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.786787 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.786745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c017d51-ac33-45e1-83a3-0c08e71053cb-serving-cert\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.786787 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:40.786775    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls podName:23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc nodeName:}" failed. No retries permitted until 2026-09-16 18:48:41.286754858 +0000 UTC m=+271.480618537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2kkqb" (UID: "23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:40.786947 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.786876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2c017d51-ac33-45e1-83a3-0c08e71053cb-snapshots\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.786947 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.786929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpql2\" (UniqueName: \"kubernetes.io/projected/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-kube-api-access-lpql2\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:40.787044 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.786959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c017d51-ac33-45e1-83a3-0c08e71053cb-trusted-ca-bundle\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.787044 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.786983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6v4\" (UniqueName: \"kubernetes.io/projected/2c017d51-ac33-45e1-83a3-0c08e71053cb-kube-api-access-sb6v4\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.787165 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.787144    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c017d51-ac33-45e1-83a3-0c08e71053cb-tmp\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.787223 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.787201    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c017d51-ac33-45e1-83a3-0c08e71053cb-service-ca-bundle\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.787359 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.787333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:40.787600 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.787577    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2c017d51-ac33-45e1-83a3-0c08e71053cb-snapshots\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.788031 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.788015    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c017d51-ac33-45e1-83a3-0c08e71053cb-trusted-ca-bundle\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.789178 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.789154    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c017d51-ac33-45e1-83a3-0c08e71053cb-serving-cert\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.798472 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.798448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpql2\" (UniqueName: \"kubernetes.io/projected/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-kube-api-access-lpql2\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:40.798665 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.798642    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6v4\" (UniqueName: \"kubernetes.io/projected/2c017d51-ac33-45e1-83a3-0c08e71053cb-kube-api-access-sb6v4\") pod \"insights-operator-5844b856db-k6nld\" (UID: \"2c017d51-ac33-45e1-83a3-0c08e71053cb\") " pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:40.807310 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.807290    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv"
Sep 16 18:48:40.830934 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:40.830900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-k6nld"
Sep 16 18:48:41.082020 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:41.081942    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv"]
Sep 16 18:48:41.085122 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:48:41.085093    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb408ab8_669f_4ae8_9e41_39ddddcfcb69.slice/crio-f241bc5f0528917851bd01bb2d01b0c550008259b80d2408bdec9c44d6ce3979 WatchSource:0}: Error finding container f241bc5f0528917851bd01bb2d01b0c550008259b80d2408bdec9c44d6ce3979: Status 404 returned error can't find the container with id f241bc5f0528917851bd01bb2d01b0c550008259b80d2408bdec9c44d6ce3979
Sep 16 18:48:41.108855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:41.108811    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-k6nld"]
Sep 16 18:48:41.112165 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:48:41.112139    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c017d51_ac33_45e1_83a3_0c08e71053cb.slice/crio-a4c30ed5b697f4d95c4c11d24567b68d70e7c2a8f8f0770136745789b4f914f8 WatchSource:0}: Error finding container a4c30ed5b697f4d95c4c11d24567b68d70e7c2a8f8f0770136745789b4f914f8: Status 404 returned error can't find the container with id a4c30ed5b697f4d95c4c11d24567b68d70e7c2a8f8f0770136745789b4f914f8
Sep 16 18:48:41.190812 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:41.190772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:41.194539 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:41.191167    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:48:41.194539 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:41.191268    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls podName:fec3d7e3-8425-4bc7-a513-f2a545a7512f nodeName:}" failed. No retries permitted until 2026-09-16 18:48:42.191246428 +0000 UTC m=+272.385110095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xdpkl" (UID: "fec3d7e3-8425-4bc7-a513-f2a545a7512f") : secret "samples-operator-tls" not found
Sep 16 18:48:41.291782 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:41.291746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:41.291976 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:41.291954    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:41.292042 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:41.292031    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls podName:23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc nodeName:}" failed. No retries permitted until 2026-09-16 18:48:42.292011168 +0000 UTC m=+272.485874843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2kkqb" (UID: "23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:41.300827 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:41.300792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv" event={"ID":"fb408ab8-669f-4ae8-9e41-39ddddcfcb69","Type":"ContainerStarted","Data":"f241bc5f0528917851bd01bb2d01b0c550008259b80d2408bdec9c44d6ce3979"}
Sep 16 18:48:41.301949 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:41.301928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-k6nld" event={"ID":"2c017d51-ac33-45e1-83a3-0c08e71053cb","Type":"ContainerStarted","Data":"a4c30ed5b697f4d95c4c11d24567b68d70e7c2a8f8f0770136745789b4f914f8"}
Sep 16 18:48:42.200141 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:42.200095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:42.200633 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:42.200286    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:48:42.200633 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:42.200363    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls podName:fec3d7e3-8425-4bc7-a513-f2a545a7512f nodeName:}" failed. No retries permitted until 2026-09-16 18:48:44.200343746 +0000 UTC m=+274.394207399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xdpkl" (UID: "fec3d7e3-8425-4bc7-a513-f2a545a7512f") : secret "samples-operator-tls" not found
Sep 16 18:48:42.300660 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:42.300618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:42.300873 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:42.300753    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:42.300873 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:42.300827    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls podName:23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc nodeName:}" failed. No retries permitted until 2026-09-16 18:48:44.300807992 +0000 UTC m=+274.494671665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2kkqb" (UID: "23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:43.311381 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:43.311316    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv" event={"ID":"fb408ab8-669f-4ae8-9e41-39ddddcfcb69","Type":"ContainerStarted","Data":"3af8d1067f3dfcda105042901f4178c0ca0b33f995b54364d469827d8286c2c8"}
Sep 16 18:48:43.313195 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:43.313167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-k6nld" event={"ID":"2c017d51-ac33-45e1-83a3-0c08e71053cb","Type":"ContainerStarted","Data":"a8803d396c83fec6c4fd215015dbb1737b4d2c5729dc4bbb5c41cb11d488227f"}
Sep 16 18:48:43.325642 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:43.325557    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7tcbv" podStartSLOduration=2.006641961 podStartE2EDuration="3.325540089s" totalImagesPullingTime="1.318898128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:43.324893009 +0000 UTC m=+273.518756747" watchObservedRunningTime="2026-09-16 18:48:43.325540089 +0000 UTC m=+273.519403761"
Sep 16 18:48:43.343448 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:43.343396    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-k6nld" podStartSLOduration=1.243464152 podStartE2EDuration="3.343380447s" totalImagesPullingTime="2.099916295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:43.342186586 +0000 UTC m=+273.536050257" watchObservedRunningTime="2026-09-16 18:48:43.343380447 +0000 UTC m=+273.537244116"
Sep 16 18:48:44.218530 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:44.218495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:44.218700 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:44.218654    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:48:44.218745 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:44.218719    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls podName:fec3d7e3-8425-4bc7-a513-f2a545a7512f nodeName:}" failed. No retries permitted until 2026-09-16 18:48:48.218703414 +0000 UTC m=+278.412567067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xdpkl" (UID: "fec3d7e3-8425-4bc7-a513-f2a545a7512f") : secret "samples-operator-tls" not found
Sep 16 18:48:44.319813 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:44.319777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:44.320206 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:44.319911    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:44.320206 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:44.319971    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls podName:23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc nodeName:}" failed. No retries permitted until 2026-09-16 18:48:48.319956802 +0000 UTC m=+278.513820474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2kkqb" (UID: "23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:45.061350 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.061309    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9kcjh"]
Sep 16 18:48:45.064619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.064596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.066751 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.066719    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:48:45.066916 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.066897    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:48:45.068046 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.068026    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lpzdr\""
Sep 16 18:48:45.076279 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.076252    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9kcjh"]
Sep 16 18:48:45.227896 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.227833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.228180 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.228157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-data-volume\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.228365 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.228350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-crio-socket\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.228526 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.228501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.228742 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.228726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvc6\" (UniqueName: \"kubernetes.io/projected/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-api-access-wgvc6\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.330008 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.329904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-crio-socket\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.330008 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.329949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.330475 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:45.330049    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:45.330475 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.330052    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-crio-socket\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.330475 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:45.330110    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls podName:6a62d4ce-d4f6-4c7c-bdc7-6fa626388458 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:45.830093231 +0000 UTC m=+276.023956887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kcjh" (UID: "6a62d4ce-d4f6-4c7c-bdc7-6fa626388458") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:45.330475 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.330153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvc6\" (UniqueName: \"kubernetes.io/projected/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-api-access-wgvc6\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.330475 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.330183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.330475 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.330208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-data-volume\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.330475 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:45.330294    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:48:45.330475 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:45.330360    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm podName:6a62d4ce-d4f6-4c7c-bdc7-6fa626388458 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:45.83034432 +0000 UTC m=+276.024207988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9kcjh" (UID: "6a62d4ce-d4f6-4c7c-bdc7-6fa626388458") : configmap "kube-rbac-proxy" not found
Sep 16 18:48:45.330763 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.330548    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-data-volume\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.340975 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.340919    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvc6\" (UniqueName: \"kubernetes.io/projected/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-api-access-wgvc6\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.835115 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.835068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.835334 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.835203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:45.835334 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:45.835261    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:45.835404 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:45.835336    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls podName:6a62d4ce-d4f6-4c7c-bdc7-6fa626388458 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:46.835319851 +0000 UTC m=+277.029183506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kcjh" (UID: "6a62d4ce-d4f6-4c7c-bdc7-6fa626388458") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:45.835404 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:45.835336    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:48:45.835476 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:45.835406    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm podName:6a62d4ce-d4f6-4c7c-bdc7-6fa626388458 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:46.835388525 +0000 UTC m=+277.029252176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9kcjh" (UID: "6a62d4ce-d4f6-4c7c-bdc7-6fa626388458") : configmap "kube-rbac-proxy" not found
Sep 16 18:48:45.863787 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.863753    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np"]
Sep 16 18:48:45.866974 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.866958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np"
Sep 16 18:48:45.869057 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.869036    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:48:45.869057 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.869042    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:48:45.869259 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.869185    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wz6ls\""
Sep 16 18:48:45.878305 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:45.878277    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np"]
Sep 16 18:48:46.036826 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:46.036790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwlj9\" (UniqueName: \"kubernetes.io/projected/ec42b10f-f217-4581-a844-135cfa4a1e53-kube-api-access-zwlj9\") pod \"migrator-8b4d697c5-ft4np\" (UID: \"ec42b10f-f217-4581-a844-135cfa4a1e53\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np"
Sep 16 18:48:46.138191 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:46.138088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwlj9\" (UniqueName: \"kubernetes.io/projected/ec42b10f-f217-4581-a844-135cfa4a1e53-kube-api-access-zwlj9\") pod \"migrator-8b4d697c5-ft4np\" (UID: \"ec42b10f-f217-4581-a844-135cfa4a1e53\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np"
Sep 16 18:48:46.145186 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:46.145164    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwlj9\" (UniqueName: \"kubernetes.io/projected/ec42b10f-f217-4581-a844-135cfa4a1e53-kube-api-access-zwlj9\") pod \"migrator-8b4d697c5-ft4np\" (UID: \"ec42b10f-f217-4581-a844-135cfa4a1e53\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np"
Sep 16 18:48:46.175910 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:46.175873    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np"
Sep 16 18:48:46.443889 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:46.443860    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np"]
Sep 16 18:48:46.447677 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:48:46.447640    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec42b10f_f217_4581_a844_135cfa4a1e53.slice/crio-ec6956eb2d66f4afb0e21b75a40a47cff9acd6c14882f19acd1117f1ac03a25f WatchSource:0}: Error finding container ec6956eb2d66f4afb0e21b75a40a47cff9acd6c14882f19acd1117f1ac03a25f: Status 404 returned error can't find the container with id ec6956eb2d66f4afb0e21b75a40a47cff9acd6c14882f19acd1117f1ac03a25f
Sep 16 18:48:46.843702 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:46.843601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:46.843919 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:46.843734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:46.843919 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:46.843768    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:46.843919 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:46.843867    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls podName:6a62d4ce-d4f6-4c7c-bdc7-6fa626388458 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:48.843826903 +0000 UTC m=+279.037690556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kcjh" (UID: "6a62d4ce-d4f6-4c7c-bdc7-6fa626388458") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:46.843919 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:46.843872    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:48:46.844092 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:46.843929    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm podName:6a62d4ce-d4f6-4c7c-bdc7-6fa626388458 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:48.843911598 +0000 UTC m=+279.037775266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9kcjh" (UID: "6a62d4ce-d4f6-4c7c-bdc7-6fa626388458") : configmap "kube-rbac-proxy" not found
Sep 16 18:48:46.935848 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:46.935806    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6rmzs_641e581f-3294-4bfa-852e-8f3bc14f54df/dns-node-resolver/0.log"
Sep 16 18:48:47.326423 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.326383    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np" event={"ID":"ec42b10f-f217-4581-a844-135cfa4a1e53","Type":"ContainerStarted","Data":"ec6956eb2d66f4afb0e21b75a40a47cff9acd6c14882f19acd1117f1ac03a25f"}
Sep 16 18:48:47.929134 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.929097    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-42c2d"]
Sep 16 18:48:47.932125 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.932107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:47.934656 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.934637    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:48:47.934771 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.934722    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:48:47.935403 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.935384    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:48:47.935403 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.935397    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mk988\""
Sep 16 18:48:47.935529 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.935406    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:48:47.952855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.952798    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-42c2d"]
Sep 16 18:48:47.953987 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.953961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/884e5fb9-694d-40a4-bac9-c80d7923da2f-signing-key\") pod \"service-ca-568869fdc5-42c2d\" (UID: \"884e5fb9-694d-40a4-bac9-c80d7923da2f\") " pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:47.954077 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.954055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/884e5fb9-694d-40a4-bac9-c80d7923da2f-signing-cabundle\") pod \"service-ca-568869fdc5-42c2d\" (UID: \"884e5fb9-694d-40a4-bac9-c80d7923da2f\") " pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:47.954153 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:47.954137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r9d\" (UniqueName: \"kubernetes.io/projected/884e5fb9-694d-40a4-bac9-c80d7923da2f-kube-api-access-64r9d\") pod \"service-ca-568869fdc5-42c2d\" (UID: \"884e5fb9-694d-40a4-bac9-c80d7923da2f\") " pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:48.054610 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.054510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r9d\" (UniqueName: \"kubernetes.io/projected/884e5fb9-694d-40a4-bac9-c80d7923da2f-kube-api-access-64r9d\") pod \"service-ca-568869fdc5-42c2d\" (UID: \"884e5fb9-694d-40a4-bac9-c80d7923da2f\") " pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:48.054784 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.054626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/884e5fb9-694d-40a4-bac9-c80d7923da2f-signing-key\") pod \"service-ca-568869fdc5-42c2d\" (UID: \"884e5fb9-694d-40a4-bac9-c80d7923da2f\") " pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:48.054784 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.054681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/884e5fb9-694d-40a4-bac9-c80d7923da2f-signing-cabundle\") pod \"service-ca-568869fdc5-42c2d\" (UID: \"884e5fb9-694d-40a4-bac9-c80d7923da2f\") " pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:48.055378 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.055354    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/884e5fb9-694d-40a4-bac9-c80d7923da2f-signing-cabundle\") pod \"service-ca-568869fdc5-42c2d\" (UID: \"884e5fb9-694d-40a4-bac9-c80d7923da2f\") " pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:48.057054 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.057035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/884e5fb9-694d-40a4-bac9-c80d7923da2f-signing-key\") pod \"service-ca-568869fdc5-42c2d\" (UID: \"884e5fb9-694d-40a4-bac9-c80d7923da2f\") " pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:48.062968 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.062942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r9d\" (UniqueName: \"kubernetes.io/projected/884e5fb9-694d-40a4-bac9-c80d7923da2f-kube-api-access-64r9d\") pod \"service-ca-568869fdc5-42c2d\" (UID: \"884e5fb9-694d-40a4-bac9-c80d7923da2f\") " pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:48.240329 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.240292    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-42c2d"
Sep 16 18:48:48.255808 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.255778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:48.255961 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:48.255944    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:48:48.256019 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:48.256009    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls podName:fec3d7e3-8425-4bc7-a513-f2a545a7512f nodeName:}" failed. No retries permitted until 2026-09-16 18:48:56.255992639 +0000 UTC m=+286.449856306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xdpkl" (UID: "fec3d7e3-8425-4bc7-a513-f2a545a7512f") : secret "samples-operator-tls" not found
Sep 16 18:48:48.333075 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.332997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np" event={"ID":"ec42b10f-f217-4581-a844-135cfa4a1e53","Type":"ContainerStarted","Data":"d45ae718df9bf0c36fdfff737e9a8cb8699c4156393c0dc48b0c4d4e12efa127"}
Sep 16 18:48:48.333075 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.333046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np" event={"ID":"ec42b10f-f217-4581-a844-135cfa4a1e53","Type":"ContainerStarted","Data":"fc3923f9a7cc92757a36adf5c0b38691d9667b50f39e95b83a16d8b1b2d848f3"}
Sep 16 18:48:48.337356 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.337338    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x4m4g_04f86fca-369a-49ce-8bd8-b734182ec3a5/node-ca/0.log"
Sep 16 18:48:48.347629 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.347584    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ft4np" podStartSLOduration=2.028291395 podStartE2EDuration="3.34756576s" totalImagesPullingTime="1.319274365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:48.3466903 +0000 UTC m=+278.540554090" watchObservedRunningTime="2026-09-16 18:48:48.34756576 +0000 UTC m=+278.541429432"
Sep 16 18:48:48.356566 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.356536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:48.356719 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:48.356699    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:48.356788 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:48.356777    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls podName:23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc nodeName:}" failed. No retries permitted until 2026-09-16 18:48:56.356757495 +0000 UTC m=+286.550621159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2kkqb" (UID: "23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:48.512593 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.512562    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-42c2d"]
Sep 16 18:48:48.515559 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:48:48.515528    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884e5fb9_694d_40a4_bac9_c80d7923da2f.slice/crio-fa22a1b7b7a3ec7b808db3520ebf6c07ab47d2893d4fba9aca9c3ea7f8a9a6d0 WatchSource:0}: Error finding container fa22a1b7b7a3ec7b808db3520ebf6c07ab47d2893d4fba9aca9c3ea7f8a9a6d0: Status 404 returned error can't find the container with id fa22a1b7b7a3ec7b808db3520ebf6c07ab47d2893d4fba9aca9c3ea7f8a9a6d0
Sep 16 18:48:48.860565 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.860525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:48.860743 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:48.860631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:48.860743 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:48.860679    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:48.860743 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:48.860704    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:48:48.860743 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:48.860743    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls podName:6a62d4ce-d4f6-4c7c-bdc7-6fa626388458 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:52.860729053 +0000 UTC m=+283.054592703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kcjh" (UID: "6a62d4ce-d4f6-4c7c-bdc7-6fa626388458") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:48.860909 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:48.860757    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm podName:6a62d4ce-d4f6-4c7c-bdc7-6fa626388458 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:52.860751008 +0000 UTC m=+283.054614657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9kcjh" (UID: "6a62d4ce-d4f6-4c7c-bdc7-6fa626388458") : configmap "kube-rbac-proxy" not found
Sep 16 18:48:49.136811 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:49.136734    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tz7mv_e68a197f-86c7-45e7-a856-4b5574581420/serve-healthcheck-canary/0.log"
Sep 16 18:48:49.335545 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:49.335496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-42c2d" event={"ID":"884e5fb9-694d-40a4-bac9-c80d7923da2f","Type":"ContainerStarted","Data":"fa22a1b7b7a3ec7b808db3520ebf6c07ab47d2893d4fba9aca9c3ea7f8a9a6d0"}
Sep 16 18:48:51.022048 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:51.022009    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh" podUID="7c727208-7c86-4cb0-ae4d-f43027417da6"
Sep 16 18:48:51.022048 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:51.022009    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb" podUID="e101c2f1-e28b-4910-8d86-9d3265060b4c"
Sep 16 18:48:51.344005 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:51.343913    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:48:51.344005 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:51.343947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-42c2d" event={"ID":"884e5fb9-694d-40a4-bac9-c80d7923da2f","Type":"ContainerStarted","Data":"aedd79b060e6d5e6337c2e4f1f7ee3babd1753e8c19d7a4b3f14b35ed69cfbb8"}
Sep 16 18:48:51.344221 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:51.344070    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:48:51.360224 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:51.360172    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-42c2d" podStartSLOduration=2.266470233 podStartE2EDuration="4.360154759s" totalImagesPullingTime="2.093684526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:51.358558203 +0000 UTC m=+281.552421886" watchObservedRunningTime="2026-09-16 18:48:51.360154759 +0000 UTC m=+281.554018431"
Sep 16 18:48:52.892228 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:52.892177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:52.892633 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:52.892323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:52.892633 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:52.892335    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:52.892633 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:52.892398    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls podName:6a62d4ce-d4f6-4c7c-bdc7-6fa626388458 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:00.89238303 +0000 UTC m=+291.086246685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kcjh" (UID: "6a62d4ce-d4f6-4c7c-bdc7-6fa626388458") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:48:52.893038 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:52.893012    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:48:54.711259 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:54.711210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") pod \"image-registry-7787c9b66b-2tlsh\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") " pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:48:54.711712 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:54.711287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:48:54.711712 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:54.711348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:48:54.711712 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:54.711389    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:48:54.711712 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:54.711414    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7787c9b66b-2tlsh: secret "image-registry-tls" not found
Sep 16 18:48:54.711712 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:54.711427    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:48:54.711712 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:54.711433    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:48:54.711712 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:54.711484    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls podName:7c727208-7c86-4cb0-ae4d-f43027417da6 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:56.711462366 +0000 UTC m=+406.905326021 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls") pod "image-registry-7787c9b66b-2tlsh" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6") : secret "image-registry-tls" not found
Sep 16 18:48:54.711712 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:54.711509    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls podName:cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc nodeName:}" failed. No retries permitted until 2026-09-16 18:50:56.711491035 +0000 UTC m=+406.905354699 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls") pod "dns-default-kv4z5" (UID: "cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc") : secret "dns-default-metrics-tls" not found
Sep 16 18:48:54.711712 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:54.711527    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert podName:e101c2f1-e28b-4910-8d86-9d3265060b4c nodeName:}" failed. No retries permitted until 2026-09-16 18:50:56.711518469 +0000 UTC m=+406.905382119 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-pkgnb" (UID: "e101c2f1-e28b-4910-8d86-9d3265060b4c") : secret "networking-console-plugin-cert" not found
Sep 16 18:48:56.325722 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:56.325681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:56.328100 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:56.328070    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec3d7e3-8425-4bc7-a513-f2a545a7512f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xdpkl\" (UID: \"fec3d7e3-8425-4bc7-a513-f2a545a7512f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:56.417355 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:56.417294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"
Sep 16 18:48:56.426721 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:56.426695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:48:56.426908 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:56.426883    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:56.426989 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:48:56.426958    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls podName:23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc nodeName:}" failed. No retries permitted until 2026-09-16 18:49:12.426941755 +0000 UTC m=+302.620805422 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2kkqb" (UID: "23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:48:56.688720 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:56.688677    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl"]
Sep 16 18:48:57.365011 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:57.364942    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl" event={"ID":"fec3d7e3-8425-4bc7-a513-f2a545a7512f","Type":"ContainerStarted","Data":"b74177296cd87fc491a17a8b7b6eaecdc4f6889fc0ff0c37ce88499e0b10f673"}
Sep 16 18:48:59.373603 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:59.373567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl" event={"ID":"fec3d7e3-8425-4bc7-a513-f2a545a7512f","Type":"ContainerStarted","Data":"45d84bce48a84633618210f778fa072e69cf5bb97d0b1e2e461c78fda6a38402"}
Sep 16 18:48:59.374001 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:59.373620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl" event={"ID":"fec3d7e3-8425-4bc7-a513-f2a545a7512f","Type":"ContainerStarted","Data":"e47bff7857fde94f4324261c9708e0d7ba7bda709307f164dbeb5267e59700d1"}
Sep 16 18:48:59.389857 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:48:59.389773    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xdpkl" podStartSLOduration=17.632001858 podStartE2EDuration="19.389751803s" totalImagesPullingTime="1.757749945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:59.389259096 +0000 UTC m=+289.583122769" watchObservedRunningTime="2026-09-16 18:48:59.389751803 +0000 UTC m=+289.583615476"
Sep 16 18:49:00.966963 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:00.966927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:49:00.969142 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:00.969123    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a62d4ce-d4f6-4c7c-bdc7-6fa626388458-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kcjh\" (UID: \"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458\") " pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:49:00.974124 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:00.974103    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9kcjh"
Sep 16 18:49:01.243248 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:01.243159    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9kcjh"]
Sep 16 18:49:01.245946 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:01.245916    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a62d4ce_d4f6_4c7c_bdc7_6fa626388458.slice/crio-d01a857b7586b4037bc3d9f121709fc909042753898ee90daf704c848807870f WatchSource:0}: Error finding container d01a857b7586b4037bc3d9f121709fc909042753898ee90daf704c848807870f: Status 404 returned error can't find the container with id d01a857b7586b4037bc3d9f121709fc909042753898ee90daf704c848807870f
Sep 16 18:49:01.381791 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:01.381758    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9kcjh" event={"ID":"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458","Type":"ContainerStarted","Data":"6069723aed9485601acc7c59b3e33c48ecda4adb0b5720b0580e44eaac43641a"}
Sep 16 18:49:01.381976 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:01.381814    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9kcjh" event={"ID":"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458","Type":"ContainerStarted","Data":"d01a857b7586b4037bc3d9f121709fc909042753898ee90daf704c848807870f"}
Sep 16 18:49:02.330797 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:49:02.330702    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fzv4f" podUID="5d7a2ef5-a033-4dd2-89e3-bc86658a3373"
Sep 16 18:49:02.386521 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:02.386489    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9kcjh" event={"ID":"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458","Type":"ContainerStarted","Data":"6550844fc75744946fc914860d5f64970ac1308d4e6b4e7275f6749689813862"}
Sep 16 18:49:03.331052 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:49:03.331007    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kv4z5" podUID="cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc"
Sep 16 18:49:04.394564 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:04.394526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9kcjh" event={"ID":"6a62d4ce-d4f6-4c7c-bdc7-6fa626388458","Type":"ContainerStarted","Data":"07d128947d0654537de16f523bf15027f00b46ae6d53fb6badf0cce4517bdbd5"}
Sep 16 18:49:04.410858 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:04.410790    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9kcjh" podStartSLOduration=17.362698196 podStartE2EDuration="19.410771556s" totalImagesPullingTime="2.04807336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:48:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:04.410630532 +0000 UTC m=+294.604494203" watchObservedRunningTime="2026-09-16 18:49:04.410771556 +0000 UTC m=+294.604635231"
Sep 16 18:49:06.699906 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:06.699870    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-zcgn2"]
Sep 16 18:49:06.703184 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:06.703161    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zcgn2"
Sep 16 18:49:06.705450 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:06.705430    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gz9nb\""
Sep 16 18:49:06.707834 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:06.707807    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:49:06.707928 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:06.707913    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:49:06.728734 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:06.728706    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zcgn2"]
Sep 16 18:49:06.813730 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:06.813698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qhb7\" (UniqueName: \"kubernetes.io/projected/b99fc008-2504-4554-a770-92b9160a3e36-kube-api-access-4qhb7\") pod \"downloads-76f99584f4-zcgn2\" (UID: \"b99fc008-2504-4554-a770-92b9160a3e36\") " pod="openshift-console/downloads-76f99584f4-zcgn2"
Sep 16 18:49:06.914904 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:06.914871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qhb7\" (UniqueName: \"kubernetes.io/projected/b99fc008-2504-4554-a770-92b9160a3e36-kube-api-access-4qhb7\") pod \"downloads-76f99584f4-zcgn2\" (UID: \"b99fc008-2504-4554-a770-92b9160a3e36\") " pod="openshift-console/downloads-76f99584f4-zcgn2"
Sep 16 18:49:06.926361 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:06.926324    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qhb7\" (UniqueName: \"kubernetes.io/projected/b99fc008-2504-4554-a770-92b9160a3e36-kube-api-access-4qhb7\") pod \"downloads-76f99584f4-zcgn2\" (UID: \"b99fc008-2504-4554-a770-92b9160a3e36\") " pod="openshift-console/downloads-76f99584f4-zcgn2"
Sep 16 18:49:07.011769 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:07.011688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zcgn2"
Sep 16 18:49:07.281207 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:07.281133    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zcgn2"]
Sep 16 18:49:07.284395 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:07.284367    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99fc008_2504_4554_a770_92b9160a3e36.slice/crio-de52fa7097d5564dc602f66d6faa9305b7c62ad002f149b6877e8abb085ac5aa WatchSource:0}: Error finding container de52fa7097d5564dc602f66d6faa9305b7c62ad002f149b6877e8abb085ac5aa: Status 404 returned error can't find the container with id de52fa7097d5564dc602f66d6faa9305b7c62ad002f149b6877e8abb085ac5aa
Sep 16 18:49:07.405053 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:07.405026    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zcgn2" event={"ID":"b99fc008-2504-4554-a770-92b9160a3e36","Type":"ContainerStarted","Data":"de52fa7097d5564dc602f66d6faa9305b7c62ad002f149b6877e8abb085ac5aa"}
Sep 16 18:49:10.253140 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:10.253108    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:49:10.257776 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:10.257747    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:49:10.258110 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:10.258090    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:49:12.464046 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:12.464012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:49:12.469237 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:12.466669    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2kkqb\" (UID: \"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:49:12.627556 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:12.627527    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8wftk\""
Sep 16 18:49:12.636136 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:12.636108    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"
Sep 16 18:49:12.915958 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:12.915857    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb"]
Sep 16 18:49:12.919412 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:12.919380    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c96a4f_7cf4_4d6e_85ad_ea75b7761fdc.slice/crio-9f8de5016384e8d53aee6cc4d74b38148d7cd20b12c24022c2ffdd038e012214 WatchSource:0}: Error finding container 9f8de5016384e8d53aee6cc4d74b38148d7cd20b12c24022c2ffdd038e012214: Status 404 returned error can't find the container with id 9f8de5016384e8d53aee6cc4d74b38148d7cd20b12c24022c2ffdd038e012214
Sep 16 18:49:12.922055 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:12.922034    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:49:13.428855 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:13.428792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb" event={"ID":"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc","Type":"ContainerStarted","Data":"9f8de5016384e8d53aee6cc4d74b38148d7cd20b12c24022c2ffdd038e012214"}
Sep 16 18:49:15.250819 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.250785    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"]
Sep 16 18:49:15.264474 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.264425    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"]
Sep 16 18:49:15.264643 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.264568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"
Sep 16 18:49:15.267936 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.267909    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-62849\""
Sep 16 18:49:15.269588 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.269516    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:49:15.389874 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.389820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b22bba5-6ae8-4bd9-9c3b-966cc2675de3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j5gsq\" (UID: \"8b22bba5-6ae8-4bd9-9c3b-966cc2675de3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"
Sep 16 18:49:15.443240 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.443165    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb" event={"ID":"23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc","Type":"ContainerStarted","Data":"895b5edd0cda3d2b97946c25d51627f3cff7f53f629eeffc85fd1236528392d3"}
Sep 16 18:49:15.461191 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.461136    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2kkqb" podStartSLOduration=33.631816962 podStartE2EDuration="35.46111976s" totalImagesPullingTime="1.829302798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:15.460095783 +0000 UTC m=+305.653959455" watchObservedRunningTime="2026-09-16 18:49:15.46111976 +0000 UTC m=+305.654983432"
Sep 16 18:49:15.491075 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.491031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b22bba5-6ae8-4bd9-9c3b-966cc2675de3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j5gsq\" (UID: \"8b22bba5-6ae8-4bd9-9c3b-966cc2675de3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"
Sep 16 18:49:15.491237 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:49:15.491198    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:49:15.491303 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:49:15.491264    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b22bba5-6ae8-4bd9-9c3b-966cc2675de3-tls-certificates podName:8b22bba5-6ae8-4bd9-9c3b-966cc2675de3 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:15.991243466 +0000 UTC m=+306.185107121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8b22bba5-6ae8-4bd9-9c3b-966cc2675de3-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-j5gsq" (UID: "8b22bba5-6ae8-4bd9-9c3b-966cc2675de3") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:49:15.996030 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:15.995996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b22bba5-6ae8-4bd9-9c3b-966cc2675de3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j5gsq\" (UID: \"8b22bba5-6ae8-4bd9-9c3b-966cc2675de3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"
Sep 16 18:49:15.996212 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:49:15.996181    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:49:15.996273 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:49:15.996264    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b22bba5-6ae8-4bd9-9c3b-966cc2675de3-tls-certificates podName:8b22bba5-6ae8-4bd9-9c3b-966cc2675de3 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:16.99623293 +0000 UTC m=+307.190096595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8b22bba5-6ae8-4bd9-9c3b-966cc2675de3-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-j5gsq" (UID: "8b22bba5-6ae8-4bd9-9c3b-966cc2675de3") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:49:17.005293 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:17.005256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b22bba5-6ae8-4bd9-9c3b-966cc2675de3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j5gsq\" (UID: \"8b22bba5-6ae8-4bd9-9c3b-966cc2675de3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"
Sep 16 18:49:17.008094 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:17.008065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b22bba5-6ae8-4bd9-9c3b-966cc2675de3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j5gsq\" (UID: \"8b22bba5-6ae8-4bd9-9c3b-966cc2675de3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"
Sep 16 18:49:17.078122 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:17.078086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"
Sep 16 18:49:17.329118 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:17.329024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:49:18.328902 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:18.328824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:49:23.474150 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:23.474108    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zcgn2" event={"ID":"b99fc008-2504-4554-a770-92b9160a3e36","Type":"ContainerStarted","Data":"9c6000537980301575bd82549fbbbfd733307d59f2696273d5b10f4cff8c7547"}
Sep 16 18:49:23.474709 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:23.474310    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-zcgn2"
Sep 16 18:49:23.475640 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:23.475609    2576 patch_prober.go:28] interesting pod/downloads-76f99584f4-zcgn2 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.18:8080/\": dial tcp 10.132.0.18:8080: connect: connection refused" start-of-body=
Sep 16 18:49:23.475743 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:23.475683    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-zcgn2" podUID="b99fc008-2504-4554-a770-92b9160a3e36" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.18:8080/\": dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 16 18:49:23.492169 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:23.492121    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-zcgn2" podStartSLOduration=1.446923725 podStartE2EDuration="17.492102119s" totalImagesPullingTime="16.045178394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:23.490556369 +0000 UTC m=+313.684420041" watchObservedRunningTime="2026-09-16 18:49:23.492102119 +0000 UTC m=+313.685965792"
Sep 16 18:49:23.532314 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:23.532239    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"]
Sep 16 18:49:23.535156 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:23.535131    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b22bba5_6ae8_4bd9_9c3b_966cc2675de3.slice/crio-34a948b8b12af40332b87a85612f6d682ceec8e979613682d220bcdcb8394f99 WatchSource:0}: Error finding container 34a948b8b12af40332b87a85612f6d682ceec8e979613682d220bcdcb8394f99: Status 404 returned error can't find the container with id 34a948b8b12af40332b87a85612f6d682ceec8e979613682d220bcdcb8394f99
Sep 16 18:49:24.481038 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:24.480996    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq" event={"ID":"8b22bba5-6ae8-4bd9-9c3b-966cc2675de3","Type":"ContainerStarted","Data":"34a948b8b12af40332b87a85612f6d682ceec8e979613682d220bcdcb8394f99"}
Sep 16 18:49:24.495052 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:24.495020    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-zcgn2"
Sep 16 18:49:25.487238 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:25.487196    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq" event={"ID":"8b22bba5-6ae8-4bd9-9c3b-966cc2675de3","Type":"ContainerStarted","Data":"51987ac6b5ecc5bf47b3afaf7059740e2c3c00dd112446de6e752e3fe0947dbb"}
Sep 16 18:49:25.503147 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:25.503082    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq" podStartSLOduration=9.165751409 podStartE2EDuration="10.50305993s" totalImagesPullingTime="1.337308521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:25.502938612 +0000 UTC m=+315.696802287" watchObservedRunningTime="2026-09-16 18:49:25.50305993 +0000 UTC m=+315.696923603"
Sep 16 18:49:26.491269 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:26.491234    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"
Sep 16 18:49:26.496428 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:26.496397    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5gsq"
Sep 16 18:49:27.315047 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.315010    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"]
Sep 16 18:49:27.349192 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.349160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"]
Sep 16 18:49:27.349363 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.349280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.351662 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.351593    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:49:27.351798 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.351735    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:49:27.352691 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.352652    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-m9kcl\""
Sep 16 18:49:27.352871 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.352743    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:49:27.492028 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.492000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.492475 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.492252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.492475 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.492402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.492475 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.492438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pms\" (UniqueName: \"kubernetes.io/projected/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-kube-api-access-h9pms\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.593073 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.592993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.593073 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.593045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pms\" (UniqueName: \"kubernetes.io/projected/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-kube-api-access-h9pms\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.593294 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.593197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.593341 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.593293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.593932 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.593905    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.596089 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.596061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.596209 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.596139    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.601072 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.601047    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pms\" (UniqueName: \"kubernetes.io/projected/8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f-kube-api-access-h9pms\") pod \"prometheus-operator-68bcd7f987-n9lq5\" (UID: \"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.661213 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.661176    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"
Sep 16 18:49:27.958244 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:27.958214    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1da3a0_ac71_47d9_8eb0_fcf5ad7b394f.slice/crio-9451206e553c48533679a767b3097b87b42e3d200b787ba71a1f9fbc3778d9f8 WatchSource:0}: Error finding container 9451206e553c48533679a767b3097b87b42e3d200b787ba71a1f9fbc3778d9f8: Status 404 returned error can't find the container with id 9451206e553c48533679a767b3097b87b42e3d200b787ba71a1f9fbc3778d9f8
Sep 16 18:49:27.961267 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:27.961242    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5"]
Sep 16 18:49:28.500849 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:28.500795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5" event={"ID":"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f","Type":"ContainerStarted","Data":"9451206e553c48533679a767b3097b87b42e3d200b787ba71a1f9fbc3778d9f8"}
Sep 16 18:49:29.153089 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.153055    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7787c9b66b-2tlsh"]
Sep 16 18:49:29.153364 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:49:29.153337    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh" podUID="7c727208-7c86-4cb0-ae4d-f43027417da6"
Sep 16 18:49:29.504591 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.504558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:49:29.525431 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.525404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:49:29.711275 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711241    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-image-registry-private-configuration\") pod \"7c727208-7c86-4cb0-ae4d-f43027417da6\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") "
Sep 16 18:49:29.711490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711284    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-certificates\") pod \"7c727208-7c86-4cb0-ae4d-f43027417da6\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") "
Sep 16 18:49:29.711490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711324    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-trusted-ca\") pod \"7c727208-7c86-4cb0-ae4d-f43027417da6\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") "
Sep 16 18:49:29.711490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711350    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-installation-pull-secrets\") pod \"7c727208-7c86-4cb0-ae4d-f43027417da6\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") "
Sep 16 18:49:29.711490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711428    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-bound-sa-token\") pod \"7c727208-7c86-4cb0-ae4d-f43027417da6\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") "
Sep 16 18:49:29.711490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711468    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c727208-7c86-4cb0-ae4d-f43027417da6-ca-trust-extracted\") pod \"7c727208-7c86-4cb0-ae4d-f43027417da6\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") "
Sep 16 18:49:29.711754 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711493    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxcqp\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-kube-api-access-bxcqp\") pod \"7c727208-7c86-4cb0-ae4d-f43027417da6\" (UID: \"7c727208-7c86-4cb0-ae4d-f43027417da6\") "
Sep 16 18:49:29.711754 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711681    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7c727208-7c86-4cb0-ae4d-f43027417da6" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:29.711754 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711702    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c727208-7c86-4cb0-ae4d-f43027417da6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7c727208-7c86-4cb0-ae4d-f43027417da6" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:29.711939 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711801    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-trusted-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:49:29.711939 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.711821    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c727208-7c86-4cb0-ae4d-f43027417da6-ca-trust-extracted\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:49:29.712061 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.712031    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7c727208-7c86-4cb0-ae4d-f43027417da6" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:29.714197 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.714124    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7c727208-7c86-4cb0-ae4d-f43027417da6" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:29.714197 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.714180    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7c727208-7c86-4cb0-ae4d-f43027417da6" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:29.714367 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.714201    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7c727208-7c86-4cb0-ae4d-f43027417da6" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:29.714367 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.714256    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-kube-api-access-bxcqp" (OuterVolumeSpecName: "kube-api-access-bxcqp") pod "7c727208-7c86-4cb0-ae4d-f43027417da6" (UID: "7c727208-7c86-4cb0-ae4d-f43027417da6"). InnerVolumeSpecName "kube-api-access-bxcqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:29.813036 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.813009    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-bound-sa-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:49:29.813157 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.813040    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxcqp\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-kube-api-access-bxcqp\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:49:29.813157 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.813057    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-image-registry-private-configuration\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:49:29.813157 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.813071    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-certificates\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:49:29.813157 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:29.813085    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c727208-7c86-4cb0-ae4d-f43027417da6-installation-pull-secrets\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:49:30.511015 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:30.510972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5" event={"ID":"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f","Type":"ContainerStarted","Data":"35ca7085191b36f01808dda263179e2d3be7ae62448b00da5be1488dcc286566"}
Sep 16 18:49:30.511547 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:30.511038    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5" event={"ID":"8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f","Type":"ContainerStarted","Data":"8da033bf14859448cbe4d3eb57707bcd5a7b7aea79c75fcdf5b0c7999210783a"}
Sep 16 18:49:30.511547 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:30.510989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7787c9b66b-2tlsh"
Sep 16 18:49:30.530656 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:30.530605    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-n9lq5" podStartSLOduration=1.74368299 podStartE2EDuration="3.530585694s" totalImagesPullingTime="1.786902704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:30.529183052 +0000 UTC m=+320.723046753" watchObservedRunningTime="2026-09-16 18:49:30.530585694 +0000 UTC m=+320.724449379"
Sep 16 18:49:30.555061 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:30.555035    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7787c9b66b-2tlsh"]
Sep 16 18:49:30.558720 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:30.558693    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7787c9b66b-2tlsh"]
Sep 16 18:49:30.721642 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:30.721600    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c727208-7c86-4cb0-ae4d-f43027417da6-registry-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:49:32.334229 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.334178    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c727208-7c86-4cb0-ae4d-f43027417da6" path="/var/lib/kubelet/pods/7c727208-7c86-4cb0-ae4d-f43027417da6/volumes"
Sep 16 18:49:32.660308 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.660268    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"]
Sep 16 18:49:32.678172 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.678142    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"]
Sep 16 18:49:32.678368 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.678298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.680793 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.680772    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sbcbf\""
Sep 16 18:49:32.681031 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.680894    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:49:32.681354 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.681175    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:49:32.688500 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.688479    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9569k"]
Sep 16 18:49:32.703329 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.703302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.706532 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.706510    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:49:32.706675 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.706531    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rc6mq\""
Sep 16 18:49:32.706737 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.706539    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:49:32.706737 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.706613    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:49:32.836479 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836449    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-tls\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.836672 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57g68\" (UniqueName: \"kubernetes.io/projected/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-kube-api-access-57g68\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.836672 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2dc7049-3be1-4518-8d6b-e40208ec0c97-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.836672 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmncz\" (UniqueName: \"kubernetes.io/projected/c2dc7049-3be1-4518-8d6b-e40208ec0c97-kube-api-access-bmncz\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.836672 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-wtmp\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.836672 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-metrics-client-ca\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.836672 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-sys\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.836968 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc7049-3be1-4518-8d6b-e40208ec0c97-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.836968 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2dc7049-3be1-4518-8d6b-e40208ec0c97-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.836968 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-root\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.836968 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-textfile\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.836968 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.836968 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.836939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.937686 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.937589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-sys\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.937686 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.937637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc7049-3be1-4518-8d6b-e40208ec0c97-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.937947 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.937702    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-sys\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.937947 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.937756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2dc7049-3be1-4518-8d6b-e40208ec0c97-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.937947 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.937795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-root\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.937947 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.937872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-textfile\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.937947 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.937903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938188 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.937953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938188 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.937906    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-root\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938188 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-tls\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938188 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57g68\" (UniqueName: \"kubernetes.io/projected/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-kube-api-access-57g68\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938188 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2dc7049-3be1-4518-8d6b-e40208ec0c97-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.938188 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmncz\" (UniqueName: \"kubernetes.io/projected/c2dc7049-3be1-4518-8d6b-e40208ec0c97-kube-api-access-bmncz\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.938490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-wtmp\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938222    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-textfile\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-metrics-client-ca\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938343    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc7049-3be1-4518-8d6b-e40208ec0c97-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.938688 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938688 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938649    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-wtmp\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.938785 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.938715    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-metrics-client-ca\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.940327 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.940297    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.940590 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.940570    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2dc7049-3be1-4518-8d6b-e40208ec0c97-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.940685 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.940664    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2dc7049-3be1-4518-8d6b-e40208ec0c97-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.940754 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.940739    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-node-exporter-tls\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.945817 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.945797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmncz\" (UniqueName: \"kubernetes.io/projected/c2dc7049-3be1-4518-8d6b-e40208ec0c97-kube-api-access-bmncz\") pod \"openshift-state-metrics-67dcf8c979-bgqdv\" (UID: \"c2dc7049-3be1-4518-8d6b-e40208ec0c97\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:32.945946 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.945929    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57g68\" (UniqueName: \"kubernetes.io/projected/9b5a216e-bf71-48a7-8781-f3aa53a97e3d-kube-api-access-57g68\") pod \"node-exporter-9569k\" (UID: \"9b5a216e-bf71-48a7-8781-f3aa53a97e3d\") " pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:32.988879 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:32.988824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"
Sep 16 18:49:33.014381 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.014350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9569k"
Sep 16 18:49:33.033442 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:33.033415    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5a216e_bf71_48a7_8781_f3aa53a97e3d.slice/crio-88f1ae790f47cc8bdf1fd6bcd9311f90a0c4f05a27d342a85234d672ca419e84 WatchSource:0}: Error finding container 88f1ae790f47cc8bdf1fd6bcd9311f90a0c4f05a27d342a85234d672ca419e84: Status 404 returned error can't find the container with id 88f1ae790f47cc8bdf1fd6bcd9311f90a0c4f05a27d342a85234d672ca419e84
Sep 16 18:49:33.280604 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.280519    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv"]
Sep 16 18:49:33.283402 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:33.283362    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2dc7049_3be1_4518_8d6b_e40208ec0c97.slice/crio-92b6c6b6f9472ad825b8b56ae5f0b3c612a686caf4a123d7d8b4a51ad6c9abcd WatchSource:0}: Error finding container 92b6c6b6f9472ad825b8b56ae5f0b3c612a686caf4a123d7d8b4a51ad6c9abcd: Status 404 returned error can't find the container with id 92b6c6b6f9472ad825b8b56ae5f0b3c612a686caf4a123d7d8b4a51ad6c9abcd
Sep 16 18:49:33.522790 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.522755    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9569k" event={"ID":"9b5a216e-bf71-48a7-8781-f3aa53a97e3d","Type":"ContainerStarted","Data":"88f1ae790f47cc8bdf1fd6bcd9311f90a0c4f05a27d342a85234d672ca419e84"}
Sep 16 18:49:33.524878 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.524827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv" event={"ID":"c2dc7049-3be1-4518-8d6b-e40208ec0c97","Type":"ContainerStarted","Data":"306fcfdc25380263788238fa7b5f884c94d43fc35a4271c67f090acaa39d9d25"}
Sep 16 18:49:33.524973 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.524907    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv" event={"ID":"c2dc7049-3be1-4518-8d6b-e40208ec0c97","Type":"ContainerStarted","Data":"9d1929dcf7504beda3ab9f734537cb48dca84002787d089773ca8d4ead4860d3"}
Sep 16 18:49:33.524973 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.524932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv" event={"ID":"c2dc7049-3be1-4518-8d6b-e40208ec0c97","Type":"ContainerStarted","Data":"92b6c6b6f9472ad825b8b56ae5f0b3c612a686caf4a123d7d8b4a51ad6c9abcd"}
Sep 16 18:49:33.757770 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.757736    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:33.770237 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.770196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.773008 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.772943    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:49:33.773008 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.772943    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:49:33.776165 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.776141    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:49:33.776505 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.776464    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:49:33.776746 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.776726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:49:33.776824 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.776756    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:49:33.777101 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.777046    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:49:33.777179 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.777137    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:49:33.777598 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.777581    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cg8q6\""
Sep 16 18:49:33.783812 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.781670    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:49:33.783812 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.783061    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:33.947449 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947644 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947644 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-config-out\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947644 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkhc\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-kube-api-access-9fkhc\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947644 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947868 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947868 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947868 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-web-config\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947868 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947868 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.947868 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-config-volume\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.948180 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:33.948180 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:33.947976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.048954 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.048858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.048954 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.048915    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.048954 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.048941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-config-out\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.048965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkhc\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-kube-api-access-9fkhc\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-web-config\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-config-volume\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.049757 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.049312    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.050030 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.050008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.052302 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.051992    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-config-out\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.053269 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.053243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.054093 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.054067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.054561 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.054537    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.054850 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.054745    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-web-config\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.054850 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.054780    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.054987 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.054961    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.055573 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.055334    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.057374 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.057332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkhc\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-kube-api-access-9fkhc\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.057465 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.057400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-config-volume\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.057725 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.057703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.082331 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.082300    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:49:34.483151 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.483120    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:49:34.517682 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:34.517651    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9faa259e_bbce_4f8a_818d_d0da36f49022.slice/crio-11caa5469ab050267afea250e672fc956284fdbf0a650c2145e1aa28d7a27eee WatchSource:0}: Error finding container 11caa5469ab050267afea250e672fc956284fdbf0a650c2145e1aa28d7a27eee: Status 404 returned error can't find the container with id 11caa5469ab050267afea250e672fc956284fdbf0a650c2145e1aa28d7a27eee
Sep 16 18:49:34.529403 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:34.529370    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerStarted","Data":"11caa5469ab050267afea250e672fc956284fdbf0a650c2145e1aa28d7a27eee"}
Sep 16 18:49:35.535676 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:35.535640    2576 generic.go:356] "Generic (PLEG): container finished" podID="9b5a216e-bf71-48a7-8781-f3aa53a97e3d" containerID="408fadeba131eb7147dc980265aff9eb01df99647a199bdd22744cfeeca076ea" exitCode=0
Sep 16 18:49:35.536183 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:35.535728    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9569k" event={"ID":"9b5a216e-bf71-48a7-8781-f3aa53a97e3d","Type":"ContainerDied","Data":"408fadeba131eb7147dc980265aff9eb01df99647a199bdd22744cfeeca076ea"}
Sep 16 18:49:35.539431 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:35.539407    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv" event={"ID":"c2dc7049-3be1-4518-8d6b-e40208ec0c97","Type":"ContainerStarted","Data":"dd5920a5001ed76a31b17c1097d643bf1021440dc18fb5c1d73f6542f9b9fad1"}
Sep 16 18:49:35.575607 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:35.575549    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bgqdv" podStartSLOduration=1.6968107959999998 podStartE2EDuration="3.57552622s" totalImagesPullingTime="1.878715424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:35.573945643 +0000 UTC m=+325.767809307" watchObservedRunningTime="2026-09-16 18:49:35.57552622 +0000 UTC m=+325.769389892"
Sep 16 18:49:36.545430 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:36.545395    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9569k" event={"ID":"9b5a216e-bf71-48a7-8781-f3aa53a97e3d","Type":"ContainerStarted","Data":"eb22591840ea777168684fd60bc8be21f389e9a8206ab8c036e3c20993d8733b"}
Sep 16 18:49:36.545773 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:36.545462    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9569k" event={"ID":"9b5a216e-bf71-48a7-8781-f3aa53a97e3d","Type":"ContainerStarted","Data":"1683a204f59c65b9bdc1fc8beb136f1511a1ad8d89fe4621fbd6c8ad6f1de3e6"}
Sep 16 18:49:36.564626 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:36.564579    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9569k" podStartSLOduration=2.974526364 podStartE2EDuration="4.564563918s" totalImagesPullingTime="1.506626841s" totalInitContainerRuntime="83.410713ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:36.563145044 +0000 UTC m=+326.757008726" watchObservedRunningTime="2026-09-16 18:49:36.564563918 +0000 UTC m=+326.758427589"
Sep 16 18:49:37.085374 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.085336    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6ccc9987d6-bhkws"]
Sep 16 18:49:37.112808 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.112778    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6ccc9987d6-bhkws"]
Sep 16 18:49:37.112978 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.112882    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.115543 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.115521    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:49:37.115631 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.115606    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xhglw\""
Sep 16 18:49:37.115673 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.115658    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:49:37.115721 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.115681    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:49:37.115982 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.115968    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-anrqm9pgdgbgn\""
Sep 16 18:49:37.116049 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.115972    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:49:37.178505 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.178186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.178505 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.178290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-metrics-server-audit-profiles\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.178505 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.178397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-secret-metrics-server-client-certs\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.178505 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.178430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-secret-metrics-server-tls\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.178505 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.178481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqtl\" (UniqueName: \"kubernetes.io/projected/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-kube-api-access-4hqtl\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.178808 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.178520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-client-ca-bundle\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.178808 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.178547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-audit-log\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.279078 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.279048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.279078 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.279083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-metrics-server-audit-profiles\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.279292 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.279129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-secret-metrics-server-client-certs\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.279292 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.279153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-secret-metrics-server-tls\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.279292 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.279193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqtl\" (UniqueName: \"kubernetes.io/projected/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-kube-api-access-4hqtl\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.279449 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.279318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-client-ca-bundle\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.279449 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.279389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-audit-log\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.279738 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.279711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-audit-log\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.279870 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.279803    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.280210 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.280183    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-metrics-server-audit-profiles\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.281679 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.281648    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-secret-metrics-server-client-certs\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.281804 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.281787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-client-ca-bundle\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.281865 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.281829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-secret-metrics-server-tls\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.286508 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.286485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqtl\" (UniqueName: \"kubernetes.io/projected/4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc-kube-api-access-4hqtl\") pod \"metrics-server-6ccc9987d6-bhkws\" (UID: \"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc\") " pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.422419 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.422374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:37.461030 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.461006    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"]
Sep 16 18:49:37.486193 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.486148    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"]
Sep 16 18:49:37.486339 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.486270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:37.488737 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.488712    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rkztr\""
Sep 16 18:49:37.488944 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.488744    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:49:37.550709 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.550680    2576 generic.go:356] "Generic (PLEG): container finished" podID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerID="397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b" exitCode=0
Sep 16 18:49:37.551097 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.550767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerDied","Data":"397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b"}
Sep 16 18:49:37.581682 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.581660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec7653b1-1d65-4e72-b3ee-001e7bfb1331-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mfnd7\" (UID: \"ec7653b1-1d65-4e72-b3ee-001e7bfb1331\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:37.582201 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.581977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2q2n\" (UniqueName: \"kubernetes.io/projected/ec7653b1-1d65-4e72-b3ee-001e7bfb1331-kube-api-access-r2q2n\") pod \"monitoring-plugin-5d85778b9d-mfnd7\" (UID: \"ec7653b1-1d65-4e72-b3ee-001e7bfb1331\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:37.682530 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.682459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2q2n\" (UniqueName: \"kubernetes.io/projected/ec7653b1-1d65-4e72-b3ee-001e7bfb1331-kube-api-access-r2q2n\") pod \"monitoring-plugin-5d85778b9d-mfnd7\" (UID: \"ec7653b1-1d65-4e72-b3ee-001e7bfb1331\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:37.682530 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.682505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec7653b1-1d65-4e72-b3ee-001e7bfb1331-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mfnd7\" (UID: \"ec7653b1-1d65-4e72-b3ee-001e7bfb1331\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:37.682743 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:49:37.682594    2576 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 18:49:37.682743 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:49:37.682668    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec7653b1-1d65-4e72-b3ee-001e7bfb1331-monitoring-plugin-cert podName:ec7653b1-1d65-4e72-b3ee-001e7bfb1331 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:38.182647316 +0000 UTC m=+328.376510967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ec7653b1-1d65-4e72-b3ee-001e7bfb1331-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-mfnd7" (UID: "ec7653b1-1d65-4e72-b3ee-001e7bfb1331") : secret "monitoring-plugin-cert" not found
Sep 16 18:49:37.691042 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.691001    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2q2n\" (UniqueName: \"kubernetes.io/projected/ec7653b1-1d65-4e72-b3ee-001e7bfb1331-kube-api-access-r2q2n\") pod \"monitoring-plugin-5d85778b9d-mfnd7\" (UID: \"ec7653b1-1d65-4e72-b3ee-001e7bfb1331\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:37.723882 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:37.723860    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6ccc9987d6-bhkws"]
Sep 16 18:49:37.725451 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:37.725425    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae43d45_2d4a_489f_8ec1_39fc6cf45ffc.slice/crio-cf496572203d49b52f8d2bc4ed94e207843400b420bebc402a6c0124cd89feaa WatchSource:0}: Error finding container cf496572203d49b52f8d2bc4ed94e207843400b420bebc402a6c0124cd89feaa: Status 404 returned error can't find the container with id cf496572203d49b52f8d2bc4ed94e207843400b420bebc402a6c0124cd89feaa
Sep 16 18:49:38.187732 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:38.187702    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec7653b1-1d65-4e72-b3ee-001e7bfb1331-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mfnd7\" (UID: \"ec7653b1-1d65-4e72-b3ee-001e7bfb1331\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:38.190546 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:38.190518    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec7653b1-1d65-4e72-b3ee-001e7bfb1331-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mfnd7\" (UID: \"ec7653b1-1d65-4e72-b3ee-001e7bfb1331\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:38.409124 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:38.409096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:38.557715 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:38.557682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws" event={"ID":"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc","Type":"ContainerStarted","Data":"cf496572203d49b52f8d2bc4ed94e207843400b420bebc402a6c0124cd89feaa"}
Sep 16 18:49:38.708596 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:38.708310    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"]
Sep 16 18:49:38.712880 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:49:38.712824    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7653b1_1d65_4e72_b3ee_001e7bfb1331.slice/crio-15e647eeb7856c1ac1f0c4049bd7c363672fffc6f121e31e27bb9253001b3751 WatchSource:0}: Error finding container 15e647eeb7856c1ac1f0c4049bd7c363672fffc6f121e31e27bb9253001b3751: Status 404 returned error can't find the container with id 15e647eeb7856c1ac1f0c4049bd7c363672fffc6f121e31e27bb9253001b3751
Sep 16 18:49:39.562874 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:39.562827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7" event={"ID":"ec7653b1-1d65-4e72-b3ee-001e7bfb1331","Type":"ContainerStarted","Data":"15e647eeb7856c1ac1f0c4049bd7c363672fffc6f121e31e27bb9253001b3751"}
Sep 16 18:49:40.571042 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:40.571007    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerStarted","Data":"616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2"}
Sep 16 18:49:40.571344 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:40.571130    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerStarted","Data":"9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd"}
Sep 16 18:49:40.571344 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:40.571174    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerStarted","Data":"36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b"}
Sep 16 18:49:40.573091 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:40.573048    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws" event={"ID":"4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc","Type":"ContainerStarted","Data":"011a3be4956aea230a917e96ec3e5fec4605172668a54914f20e5f21a734803c"}
Sep 16 18:49:40.591737 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:40.591676    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws" podStartSLOduration=1.192713621 podStartE2EDuration="3.591661695s" totalImagesPullingTime="2.398948074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:40.5902883 +0000 UTC m=+330.784151972" watchObservedRunningTime="2026-09-16 18:49:40.591661695 +0000 UTC m=+330.785525398"
Sep 16 18:49:41.577888 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:41.577852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7" event={"ID":"ec7653b1-1d65-4e72-b3ee-001e7bfb1331","Type":"ContainerStarted","Data":"da4cd75dd303e799b941dbc4ea19636a45cbd427c177b8b64f818881dce3b244"}
Sep 16 18:49:41.578292 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:41.577915    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:41.581218 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:41.581192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerStarted","Data":"8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0"}
Sep 16 18:49:41.581333 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:41.581254    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerStarted","Data":"ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7"}
Sep 16 18:49:41.582857 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:41.582820    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7"
Sep 16 18:49:41.593813 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:41.593768    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mfnd7" podStartSLOduration=1.863559178 podStartE2EDuration="4.593733899s" totalImagesPullingTime="2.730174721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:41.592943854 +0000 UTC m=+331.786807538" watchObservedRunningTime="2026-09-16 18:49:41.593733899 +0000 UTC m=+331.787597570"
Sep 16 18:49:42.588286 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:42.588209    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerStarted","Data":"b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c"}
Sep 16 18:49:42.614406 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:42.614346    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.968676134 podStartE2EDuration="9.614323937s" totalImagesPullingTime="5.632850837s" totalInitContainerRuntime="12.796966ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:42.613025054 +0000 UTC m=+332.806888763" watchObservedRunningTime="2026-09-16 18:49:42.614323937 +0000 UTC m=+332.808187609"
Sep 16 18:49:54.635365 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:54.635278    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c017d51-ac33-45e1-83a3-0c08e71053cb" containerID="a8803d396c83fec6c4fd215015dbb1737b4d2c5729dc4bbb5c41cb11d488227f" exitCode=0
Sep 16 18:49:54.635365 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:54.635352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-k6nld" event={"ID":"2c017d51-ac33-45e1-83a3-0c08e71053cb","Type":"ContainerDied","Data":"a8803d396c83fec6c4fd215015dbb1737b4d2c5729dc4bbb5c41cb11d488227f"}
Sep 16 18:49:54.635769 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:54.635670    2576 scope.go:117] "RemoveContainer" containerID="a8803d396c83fec6c4fd215015dbb1737b4d2c5729dc4bbb5c41cb11d488227f"
Sep 16 18:49:55.648634 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:55.648601    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-k6nld" event={"ID":"2c017d51-ac33-45e1-83a3-0c08e71053cb","Type":"ContainerStarted","Data":"ef8f19a2d26eb38d895f2e6f9dcd249444d630b2d0dde5745b7676f9cd275d95"}
Sep 16 18:49:57.422722 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:57.422685    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:49:57.423241 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:49:57.422780    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:50:17.427793 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:17.427723    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:50:17.431650 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:17.431628    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6ccc9987d6-bhkws"
Sep 16 18:50:23.376762 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:23.376176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:50:23.380875 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:23.380829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7a2ef5-a033-4dd2-89e3-bc86658a3373-metrics-certs\") pod \"network-metrics-daemon-fzv4f\" (UID: \"5d7a2ef5-a033-4dd2-89e3-bc86658a3373\") " pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:50:23.632352 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:23.632275    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7prs7\""
Sep 16 18:50:23.640960 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:23.640938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fzv4f"
Sep 16 18:50:23.911782 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:23.911751    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fzv4f"]
Sep 16 18:50:23.914943 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:50:23.914911    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7a2ef5_a033_4dd2_89e3_bc86658a3373.slice/crio-b37aebd3505adbbcbc1128463a03c92b8c80f11f90316f681ae3e3e2def38dde WatchSource:0}: Error finding container b37aebd3505adbbcbc1128463a03c92b8c80f11f90316f681ae3e3e2def38dde: Status 404 returned error can't find the container with id b37aebd3505adbbcbc1128463a03c92b8c80f11f90316f681ae3e3e2def38dde
Sep 16 18:50:24.759173 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:24.759138    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fzv4f" event={"ID":"5d7a2ef5-a033-4dd2-89e3-bc86658a3373","Type":"ContainerStarted","Data":"b37aebd3505adbbcbc1128463a03c92b8c80f11f90316f681ae3e3e2def38dde"}
Sep 16 18:50:25.764730 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:25.764698    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fzv4f" event={"ID":"5d7a2ef5-a033-4dd2-89e3-bc86658a3373","Type":"ContainerStarted","Data":"cc85b9d3c477155ea67ec318290994bdb79f759e32043fd12306f65153e7a223"}
Sep 16 18:50:25.765099 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:25.764752    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fzv4f" event={"ID":"5d7a2ef5-a033-4dd2-89e3-bc86658a3373","Type":"ContainerStarted","Data":"86d9efd95725330b15acdd9fbbb39b13b7e9d8c4a502f2c0609c22131e5d99a4"}
Sep 16 18:50:25.782089 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:25.782046    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fzv4f" podStartSLOduration=374.468483892 podStartE2EDuration="6m15.782030662s" totalImagesPullingTime="1.31354677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:25.780366551 +0000 UTC m=+375.974230223" watchObservedRunningTime="2026-09-16 18:50:25.782030662 +0000 UTC m=+375.975894342"
Sep 16 18:50:53.007194 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.007162    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:50:53.007569 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.007455    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="alertmanager" containerID="cri-o://36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b" gracePeriod=120
Sep 16 18:50:53.007569 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.007502    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy-metric" containerID="cri-o://8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0" gracePeriod=120
Sep 16 18:50:53.007569 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.007531    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="prom-label-proxy" containerID="cri-o://b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c" gracePeriod=120
Sep 16 18:50:53.007719 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.007571    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy-web" containerID="cri-o://616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2" gracePeriod=120
Sep 16 18:50:53.007719 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.007601    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="config-reloader" containerID="cri-o://9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd" gracePeriod=120
Sep 16 18:50:53.007719 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.007615    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy" containerID="cri-o://ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7" gracePeriod=120
Sep 16 18:50:53.873794 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.873755    2576 generic.go:356] "Generic (PLEG): container finished" podID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerID="b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c" exitCode=0
Sep 16 18:50:53.873794 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.873780    2576 generic.go:356] "Generic (PLEG): container finished" podID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerID="ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7" exitCode=0
Sep 16 18:50:53.873794 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.873786    2576 generic.go:356] "Generic (PLEG): container finished" podID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerID="9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd" exitCode=0
Sep 16 18:50:53.873794 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.873792    2576 generic.go:356] "Generic (PLEG): container finished" podID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerID="36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b" exitCode=0
Sep 16 18:50:53.874067 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.873847    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerDied","Data":"b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c"}
Sep 16 18:50:53.874067 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.873913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerDied","Data":"ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7"}
Sep 16 18:50:53.874067 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.873940    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerDied","Data":"9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd"}
Sep 16 18:50:53.874067 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:53.873965    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerDied","Data":"36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b"}
Sep 16 18:50:54.241051 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.241030    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:54.334755 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.334725    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.334755 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.334755    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-web-config\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335009 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.334778    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-metrics-client-ca\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335009 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.334824    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-main-db\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335009 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.334872    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-tls-assets\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335009 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.334980    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fkhc\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-kube-api-access-9fkhc\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335268 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335037    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335268 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335066    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-config-volume\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335268 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335109    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-main-tls\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335268 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335150    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-web\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335268 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335148    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:54.335268 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335175    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-cluster-tls-config\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335268 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335202    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:54.335268 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335223    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-config-out\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335268 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335257    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-trusted-ca-bundle\") pod \"9faa259e-bbce-4f8a-818d-d0da36f49022\" (UID: \"9faa259e-bbce-4f8a-818d-d0da36f49022\") "
Sep 16 18:50:54.335708 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335506    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-metrics-client-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.335708 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335523    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-main-db\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.336116 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.335904    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:54.337684 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.337636    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:54.338096 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.338062    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:54.338275 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.338242    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-config-volume" (OuterVolumeSpecName: "config-volume") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:54.338659 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.338627    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:54.338742 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.338707    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:54.339236 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.339212    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-config-out" (OuterVolumeSpecName: "config-out") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:54.339407 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.339389    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:54.339608 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.339584    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-kube-api-access-9fkhc" (OuterVolumeSpecName: "kube-api-access-9fkhc") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "kube-api-access-9fkhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:54.343296 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.343176    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:54.344427 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:50:54.344388    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb" podUID="e101c2f1-e28b-4910-8d86-9d3265060b4c"
Sep 16 18:50:54.348912 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.348889    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-web-config" (OuterVolumeSpecName: "web-config") pod "9faa259e-bbce-4f8a-818d-d0da36f49022" (UID: "9faa259e-bbce-4f8a-818d-d0da36f49022"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:54.436472 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436407    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436472 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436432    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-web-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436472 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436441    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-tls-assets\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436472 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436450    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fkhc\" (UniqueName: \"kubernetes.io/projected/9faa259e-bbce-4f8a-818d-d0da36f49022-kube-api-access-9fkhc\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436472 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436458    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436472 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436467    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436472 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436475    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-main-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436775 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436483    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436775 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436499    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9faa259e-bbce-4f8a-818d-d0da36f49022-cluster-tls-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436775 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436508    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9faa259e-bbce-4f8a-818d-d0da36f49022-config-out\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.436775 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.436519    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9faa259e-bbce-4f8a-818d-d0da36f49022-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.880359 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.880280    2576 generic.go:356] "Generic (PLEG): container finished" podID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerID="8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0" exitCode=0
Sep 16 18:50:54.880359 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.880304    2576 generic.go:356] "Generic (PLEG): container finished" podID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerID="616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2" exitCode=0
Sep 16 18:50:54.880677 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.880362    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:54.880677 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.880369    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerDied","Data":"8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0"}
Sep 16 18:50:54.880677 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.880456    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerDied","Data":"616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2"}
Sep 16 18:50:54.880677 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.880507    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9faa259e-bbce-4f8a-818d-d0da36f49022","Type":"ContainerDied","Data":"11caa5469ab050267afea250e672fc956284fdbf0a650c2145e1aa28d7a27eee"}
Sep 16 18:50:54.880677 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.880539    2576 scope.go:117] "RemoveContainer" containerID="b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c"
Sep 16 18:50:54.880929 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.880768    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:50:54.890112 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.890092    2576 scope.go:117] "RemoveContainer" containerID="8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0"
Sep 16 18:50:54.896871 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.896858    2576 scope.go:117] "RemoveContainer" containerID="ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7"
Sep 16 18:50:54.903503 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.903488    2576 scope.go:117] "RemoveContainer" containerID="616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2"
Sep 16 18:50:54.905514 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.905494    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:50:54.910340 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.910320    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:50:54.911061 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.911043    2576 scope.go:117] "RemoveContainer" containerID="9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd"
Sep 16 18:50:54.917584 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.917570    2576 scope.go:117] "RemoveContainer" containerID="36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b"
Sep 16 18:50:54.924232 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.924215    2576 scope.go:117] "RemoveContainer" containerID="397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b"
Sep 16 18:50:54.931030 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.931015    2576 scope.go:117] "RemoveContainer" containerID="b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c"
Sep 16 18:50:54.931279 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:50:54.931262    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c\": container with ID starting with b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c not found: ID does not exist" containerID="b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c"
Sep 16 18:50:54.931326 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.931289    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c"} err="failed to get container status \"b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c\": rpc error: code = NotFound desc = could not find container \"b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c\": container with ID starting with b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c not found: ID does not exist"
Sep 16 18:50:54.931326 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.931317    2576 scope.go:117] "RemoveContainer" containerID="8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0"
Sep 16 18:50:54.931582 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:50:54.931564    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0\": container with ID starting with 8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0 not found: ID does not exist" containerID="8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0"
Sep 16 18:50:54.931633 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.931586    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0"} err="failed to get container status \"8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0\": rpc error: code = NotFound desc = could not find container \"8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0\": container with ID starting with 8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0 not found: ID does not exist"
Sep 16 18:50:54.931633 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.931603    2576 scope.go:117] "RemoveContainer" containerID="ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7"
Sep 16 18:50:54.931853 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:50:54.931820    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7\": container with ID starting with ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7 not found: ID does not exist" containerID="ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7"
Sep 16 18:50:54.931935 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.931919    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7"} err="failed to get container status \"ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7\": rpc error: code = NotFound desc = could not find container \"ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7\": container with ID starting with ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7 not found: ID does not exist"
Sep 16 18:50:54.931970 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.931936    2576 scope.go:117] "RemoveContainer" containerID="616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2"
Sep 16 18:50:54.932156 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:50:54.932138    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2\": container with ID starting with 616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2 not found: ID does not exist" containerID="616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2"
Sep 16 18:50:54.932196 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.932161    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2"} err="failed to get container status \"616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2\": rpc error: code = NotFound desc = could not find container \"616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2\": container with ID starting with 616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2 not found: ID does not exist"
Sep 16 18:50:54.932196 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.932178    2576 scope.go:117] "RemoveContainer" containerID="9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd"
Sep 16 18:50:54.932402 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:50:54.932386    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd\": container with ID starting with 9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd not found: ID does not exist" containerID="9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd"
Sep 16 18:50:54.932440 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.932405    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd"} err="failed to get container status \"9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd\": rpc error: code = NotFound desc = could not find container \"9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd\": container with ID starting with 9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd not found: ID does not exist"
Sep 16 18:50:54.932440 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.932416    2576 scope.go:117] "RemoveContainer" containerID="36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b"
Sep 16 18:50:54.932618 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:50:54.932602    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b\": container with ID starting with 36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b not found: ID does not exist" containerID="36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b"
Sep 16 18:50:54.932659 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.932621    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b"} err="failed to get container status \"36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b\": rpc error: code = NotFound desc = could not find container \"36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b\": container with ID starting with 36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b not found: ID does not exist"
Sep 16 18:50:54.932659 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.932632    2576 scope.go:117] "RemoveContainer" containerID="397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b"
Sep 16 18:50:54.932831 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:50:54.932817    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b\": container with ID starting with 397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b not found: ID does not exist" containerID="397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b"
Sep 16 18:50:54.932894 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.932833    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b"} err="failed to get container status \"397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b\": rpc error: code = NotFound desc = could not find container \"397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b\": container with ID starting with 397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b not found: ID does not exist"
Sep 16 18:50:54.932894 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.932861    2576 scope.go:117] "RemoveContainer" containerID="b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c"
Sep 16 18:50:54.933090 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933073    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c"} err="failed to get container status \"b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c\": rpc error: code = NotFound desc = could not find container \"b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c\": container with ID starting with b9fd56a7e0745f2c3370c75ceffbea4d1b4ba31ceb662528cc37c480590c780c not found: ID does not exist"
Sep 16 18:50:54.933133 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933090    2576 scope.go:117] "RemoveContainer" containerID="8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0"
Sep 16 18:50:54.933313 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933298    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0"} err="failed to get container status \"8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0\": rpc error: code = NotFound desc = could not find container \"8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0\": container with ID starting with 8a7b5e673ab26b394f0cd1affe10942fdda839d317aa0bf12b7aca9e6e91cbe0 not found: ID does not exist"
Sep 16 18:50:54.933313 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933312    2576 scope.go:117] "RemoveContainer" containerID="ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7"
Sep 16 18:50:54.933509 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933494    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7"} err="failed to get container status \"ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7\": rpc error: code = NotFound desc = could not find container \"ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7\": container with ID starting with ca3db931c7b945a6378c7801eb12e16bbf64aab00523571090d48df3803aa5d7 not found: ID does not exist"
Sep 16 18:50:54.933550 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933508    2576 scope.go:117] "RemoveContainer" containerID="616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2"
Sep 16 18:50:54.933699 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933682    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2"} err="failed to get container status \"616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2\": rpc error: code = NotFound desc = could not find container \"616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2\": container with ID starting with 616f91fb7085d04ac9d9aa3dfe2306f2c4c19130ca65da9ba276edaab58747b2 not found: ID does not exist"
Sep 16 18:50:54.933738 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933699    2576 scope.go:117] "RemoveContainer" containerID="9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd"
Sep 16 18:50:54.933944 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933926    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd"} err="failed to get container status \"9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd\": rpc error: code = NotFound desc = could not find container \"9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd\": container with ID starting with 9c5048db9e47cbde6ce8548c09011e3c6dacb5af1c042f61eec820327c7daabd not found: ID does not exist"
Sep 16 18:50:54.933944 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.933943    2576 scope.go:117] "RemoveContainer" containerID="36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b"
Sep 16 18:50:54.934153 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.934131    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b"} err="failed to get container status \"36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b\": rpc error: code = NotFound desc = could not find container \"36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b\": container with ID starting with 36c08b67178048b14eef822feb9988aec8273bfacbca67ed36ee9505562f1f1b not found: ID does not exist"
Sep 16 18:50:54.934203 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.934153    2576 scope.go:117] "RemoveContainer" containerID="397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b"
Sep 16 18:50:54.934383 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.934367    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b"} err="failed to get container status \"397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b\": rpc error: code = NotFound desc = could not find container \"397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b\": container with ID starting with 397ea08e051d2261010694b016291425eb17b3546f9c508648e0b1528c80838b not found: ID does not exist"
Sep 16 18:50:54.937394 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937372    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:50:54.937661 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937649    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy-metric"
Sep 16 18:50:54.937710 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937663    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy-metric"
Sep 16 18:50:54.937710 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937672    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="alertmanager"
Sep 16 18:50:54.937710 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937678    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="alertmanager"
Sep 16 18:50:54.937710 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937694    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="prom-label-proxy"
Sep 16 18:50:54.937710 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937700    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="prom-label-proxy"
Sep 16 18:50:54.937710 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937707    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="config-reloader"
Sep 16 18:50:54.937710 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937712    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="config-reloader"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937718    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy-web"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937723    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy-web"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937731    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937736    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937742    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="init-config-reloader"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937747    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="init-config-reloader"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937803    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy-web"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937809    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937816    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="kube-rbac-proxy-metric"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937821    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="prom-label-proxy"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937826    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="config-reloader"
Sep 16 18:50:54.937953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.937832    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" containerName="alertmanager"
Sep 16 18:50:54.942810 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.942795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:54.945049 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.945027    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:50:54.945145 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.945058    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:50:54.945328 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.945311    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:50:54.945445 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.945427    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:50:54.945537 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.945467    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:50:54.945537 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.945474    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:50:54.945537 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.945481    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cg8q6\""
Sep 16 18:50:54.945755 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.945740    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:50:54.945901 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.945827    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:50:54.951807 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.951752    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:50:54.956404 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:54.956382    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:50:55.041529 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041656 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041656 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041656 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-web-config\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041656 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041783 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041783 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041783 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041783 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041927 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6rd\" (UniqueName: \"kubernetes.io/projected/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-kube-api-access-fk6rd\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041927 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-config-out\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041927 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.041927 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.041906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.142851 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.142770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6rd\" (UniqueName: \"kubernetes.io/projected/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-kube-api-access-fk6rd\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.142851 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.142807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-config-out\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.142851 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.142828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143106 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.142888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143106 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.142908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143106 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.142927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143106 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.142946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143106 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.142971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-web-config\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143106 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.143004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143106 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.143036    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143106 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.143057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143106 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.143079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143589 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.143115    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.143743 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.143723    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.144027 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.143999    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.145742 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.145692    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.145933 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.145885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.145933 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.145894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-config-out\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.146098 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.146035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.146248 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.146230    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.146528 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.146510    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.146619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.146538    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.146619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.146563    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.146871 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.146832    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.147771 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.147757    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-web-config\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.150580 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.150564    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6rd\" (UniqueName: \"kubernetes.io/projected/eda1ff5a-a3d7-474c-98e1-60101b05ef6f-kube-api-access-fk6rd\") pod \"alertmanager-main-0\" (UID: \"eda1ff5a-a3d7-474c-98e1-60101b05ef6f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.254879 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.254824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:50:55.536143 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.536119    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:50:55.538616 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:50:55.538591    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda1ff5a_a3d7_474c_98e1_60101b05ef6f.slice/crio-20354bb8bd7dcd71c1f03eea195656ef4876e2a09577043fa100684e281941aa WatchSource:0}: Error finding container 20354bb8bd7dcd71c1f03eea195656ef4876e2a09577043fa100684e281941aa: Status 404 returned error can't find the container with id 20354bb8bd7dcd71c1f03eea195656ef4876e2a09577043fa100684e281941aa
Sep 16 18:50:55.885822 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.885737    2576 generic.go:356] "Generic (PLEG): container finished" podID="eda1ff5a-a3d7-474c-98e1-60101b05ef6f" containerID="2f9f04fdf0c979983216ed1cfc13d9a9ba142761f0979db0c25c7995f3c7180f" exitCode=0
Sep 16 18:50:55.886008 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.885817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eda1ff5a-a3d7-474c-98e1-60101b05ef6f","Type":"ContainerDied","Data":"2f9f04fdf0c979983216ed1cfc13d9a9ba142761f0979db0c25c7995f3c7180f"}
Sep 16 18:50:55.886008 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:55.885924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eda1ff5a-a3d7-474c-98e1-60101b05ef6f","Type":"ContainerStarted","Data":"20354bb8bd7dcd71c1f03eea195656ef4876e2a09577043fa100684e281941aa"}
Sep 16 18:50:56.334193 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.334162    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9faa259e-bbce-4f8a-818d-d0da36f49022" path="/var/lib/kubelet/pods/9faa259e-bbce-4f8a-818d-d0da36f49022/volumes"
Sep 16 18:50:56.755854 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.755803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:50:56.756054 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.755867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:50:56.758186 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.758159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc-metrics-tls\") pod \"dns-default-kv4z5\" (UID: \"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc\") " pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:50:56.758289 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.758237    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e101c2f1-e28b-4910-8d86-9d3265060b4c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-pkgnb\" (UID: \"e101c2f1-e28b-4910-8d86-9d3265060b4c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:50:56.894101 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.894062    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eda1ff5a-a3d7-474c-98e1-60101b05ef6f","Type":"ContainerStarted","Data":"1fb889e22cb28678b075fede06533546bca873cb6056c251a812f1a551b29bf2"}
Sep 16 18:50:56.894255 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.894182    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eda1ff5a-a3d7-474c-98e1-60101b05ef6f","Type":"ContainerStarted","Data":"6e3ee566e1b434d8c3e15edab8d5b285c096c682dcf13d3197ff83c8dee91f49"}
Sep 16 18:50:56.894255 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.894224    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eda1ff5a-a3d7-474c-98e1-60101b05ef6f","Type":"ContainerStarted","Data":"7436b6cd175b5c577ffd1cdbb7fc6e86c3a8ae3e1262837872bdae14eae00dea"}
Sep 16 18:50:56.894396 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.894265    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eda1ff5a-a3d7-474c-98e1-60101b05ef6f","Type":"ContainerStarted","Data":"db61420839a3ac08c8b0748e1d30da41a9c3ce1ff281ef8a3006f31f1626e61d"}
Sep 16 18:50:56.894396 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.894306    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eda1ff5a-a3d7-474c-98e1-60101b05ef6f","Type":"ContainerStarted","Data":"94604ff602edd6ca03a5ce101993309869afcfcfd82e9613cea3c8935cb45784"}
Sep 16 18:50:56.894396 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.894345    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eda1ff5a-a3d7-474c-98e1-60101b05ef6f","Type":"ContainerStarted","Data":"cbc290b70b06a48c368b5b88ac1aac406ba9eb89f007e9f18ddacb6774d94650"}
Sep 16 18:50:56.920576 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.920531    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.905620812 podStartE2EDuration="2.920517966s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.897154ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:56.919461376 +0000 UTC m=+407.113325047" watchObservedRunningTime="2026-09-16 18:50:56.920517966 +0000 UTC m=+407.114381638"
Sep 16 18:50:56.983693 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.983667    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t7mvb\""
Sep 16 18:50:56.992579 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:56.992555    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"
Sep 16 18:50:57.031526 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.031498    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l7s4t\""
Sep 16 18:50:57.039928 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.039897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:50:57.079571 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.079534    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-664f4c4c65-jfztk"]
Sep 16 18:50:57.085879 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.085832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.087978 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.087954    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:50:57.087978 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.087973    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:50:57.088146 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.087977    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:50:57.088347 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.088331    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:50:57.088420 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.088399    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t8dkd\""
Sep 16 18:50:57.088585 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.088561    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:50:57.095486 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.095310    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:50:57.095603 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.095584    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-664f4c4c65-jfztk"]
Sep 16 18:50:57.159395 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.159359    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tq8k\" (UniqueName: \"kubernetes.io/projected/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-kube-api-access-5tq8k\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.159559 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.159409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-federate-client-tls\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.159559 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.159438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-telemeter-client-tls\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.159559 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.159465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-serving-certs-ca-bundle\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.159744 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.159582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-secret-telemeter-client\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.159744 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.159649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-metrics-client-ca\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.159744 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.159682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.159744 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.159732    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.260502 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.260420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tq8k\" (UniqueName: \"kubernetes.io/projected/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-kube-api-access-5tq8k\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.260502 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.260454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-federate-client-tls\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.260502 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.260477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-telemeter-client-tls\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.260765 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.260504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-serving-certs-ca-bundle\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.260765 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.260571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-secret-telemeter-client\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.260765 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.260639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-metrics-client-ca\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.260765 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.260675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.260765 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.260739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.261945 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.261915    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-serving-certs-ca-bundle\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.263558 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.263438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-metrics-client-ca\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.263680 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.263570    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-secret-telemeter-client\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.263680 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.263622    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-federate-client-tls\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.264039 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.264018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.264458 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.264436    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.265704 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.265681    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-telemeter-client-tls\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.268047 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.268027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tq8k\" (UniqueName: \"kubernetes.io/projected/2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d-kube-api-access-5tq8k\") pod \"telemeter-client-664f4c4c65-jfztk\" (UID: \"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d\") " pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.274210 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.274191    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb"]
Sep 16 18:50:57.277087 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:50:57.277067    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode101c2f1_e28b_4910_8d86_9d3265060b4c.slice/crio-8eacbfd6d7b48da081cd93f4cccf81d27a630ccadeda0a8c6dcf38324f89099b WatchSource:0}: Error finding container 8eacbfd6d7b48da081cd93f4cccf81d27a630ccadeda0a8c6dcf38324f89099b: Status 404 returned error can't find the container with id 8eacbfd6d7b48da081cd93f4cccf81d27a630ccadeda0a8c6dcf38324f89099b
Sep 16 18:50:57.343544 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.343480    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kv4z5"]
Sep 16 18:50:57.345818 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:50:57.345791    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc7ecbe_3313_46a8_a6cf_efb1c6b260fc.slice/crio-dd89ec98186976425856b68349c47a4e8e66d7250fb1f4a747a21e3d91ea2dd4 WatchSource:0}: Error finding container dd89ec98186976425856b68349c47a4e8e66d7250fb1f4a747a21e3d91ea2dd4: Status 404 returned error can't find the container with id dd89ec98186976425856b68349c47a4e8e66d7250fb1f4a747a21e3d91ea2dd4
Sep 16 18:50:57.405927 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.405896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk"
Sep 16 18:50:57.691999 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.691380    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-664f4c4c65-jfztk"]
Sep 16 18:50:57.696055 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:50:57.696022    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1bb671_f6d3_4c3c_b197_08ceb53d0d8d.slice/crio-eadcf75a1f43982b7976b66c032b8222c58e78bf5a440e193cc084f3da77ae25 WatchSource:0}: Error finding container eadcf75a1f43982b7976b66c032b8222c58e78bf5a440e193cc084f3da77ae25: Status 404 returned error can't find the container with id eadcf75a1f43982b7976b66c032b8222c58e78bf5a440e193cc084f3da77ae25
Sep 16 18:50:57.902605 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.902559    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk" event={"ID":"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d","Type":"ContainerStarted","Data":"eadcf75a1f43982b7976b66c032b8222c58e78bf5a440e193cc084f3da77ae25"}
Sep 16 18:50:57.904191 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.904128    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kv4z5" event={"ID":"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc","Type":"ContainerStarted","Data":"dd89ec98186976425856b68349c47a4e8e66d7250fb1f4a747a21e3d91ea2dd4"}
Sep 16 18:50:57.905897 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:57.905827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb" event={"ID":"e101c2f1-e28b-4910-8d86-9d3265060b4c","Type":"ContainerStarted","Data":"8eacbfd6d7b48da081cd93f4cccf81d27a630ccadeda0a8c6dcf38324f89099b"}
Sep 16 18:50:59.917960 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:59.917932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk" event={"ID":"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d","Type":"ContainerStarted","Data":"4642c87376156d90a23e95c04543187ea6ad463a04214e5df2bb2467fb3dbdcb"}
Sep 16 18:50:59.918283 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:59.918000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk" event={"ID":"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d","Type":"ContainerStarted","Data":"9c392df50f95ef5be33c8508765e3c2831a674d6e6bd33e7ab903e6c05f46e34"}
Sep 16 18:50:59.920041 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:59.920013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kv4z5" event={"ID":"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc","Type":"ContainerStarted","Data":"b45f8538744d97930bf0a278730a39cfee667301d4d937c5da2024785bd01e5f"}
Sep 16 18:50:59.920147 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:59.920074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kv4z5" event={"ID":"cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc","Type":"ContainerStarted","Data":"31d9c11e146f3661b9af626550042627fbf29ea484f6cfd2532589d36d6bf70b"}
Sep 16 18:50:59.920147 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:59.920139    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:50:59.921830 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:59.921793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb" event={"ID":"e101c2f1-e28b-4910-8d86-9d3265060b4c","Type":"ContainerStarted","Data":"bb01468b1cc85725c0878e1813fe0eb4c2cdcbe9addd2953057f3f2724c10de8"}
Sep 16 18:50:59.937767 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:59.937719    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kv4z5" podStartSLOduration=374.349775457 podStartE2EDuration="6m15.937706177s" totalImagesPullingTime="1.58793072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:59.937208403 +0000 UTC m=+410.131072076" watchObservedRunningTime="2026-09-16 18:50:59.937706177 +0000 UTC m=+410.131569880"
Sep 16 18:50:59.953691 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:50:59.953651    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-pkgnb" podStartSLOduration=378.303318941 podStartE2EDuration="6m19.953637245s" totalImagesPullingTime="1.650318304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:59.952930059 +0000 UTC m=+410.146793731" watchObservedRunningTime="2026-09-16 18:50:59.953637245 +0000 UTC m=+410.147500929"
Sep 16 18:51:00.928300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:51:00.928261    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk" event={"ID":"2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d","Type":"ContainerStarted","Data":"e93a28bd385e660535aad8052c9db7258f705489382f3a41d4f9ca23408bab15"}
Sep 16 18:51:00.950235 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:51:00.950188    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-664f4c4c65-jfztk" podStartSLOduration=1.855565839 podStartE2EDuration="3.950174104s" totalImagesPullingTime="2.094608265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:00.949810932 +0000 UTC m=+411.143674603" watchObservedRunningTime="2026-09-16 18:51:00.950174104 +0000 UTC m=+411.144037776"
Sep 16 18:51:09.931031 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:51:09.931001    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kv4z5"
Sep 16 18:54:10.287696 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:10.287667    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:54:10.290753 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:10.290734    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:54:55.880329 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:55.880248    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"]
Sep 16 18:54:55.883524 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:55.883509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:54:55.887019 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:55.886989    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:54:55.887019 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:55.887016    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:54:55.887199 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:55.886994    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-ltnfj\""
Sep 16 18:54:55.887300 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:55.887287    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:54:55.901700 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:55.901676    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"]
Sep 16 18:54:55.929031 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:55.929003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8sd4\" (UniqueName: \"kubernetes.io/projected/5768c384-a5d0-4d9c-838c-4644e42b4445-kube-api-access-x8sd4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-trbpt\" (UID: \"5768c384-a5d0-4d9c-838c-4644e42b4445\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:54:55.929185 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:55.929045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5768c384-a5d0-4d9c-838c-4644e42b4445-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-trbpt\" (UID: \"5768c384-a5d0-4d9c-838c-4644e42b4445\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:54:56.030257 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:56.030226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5768c384-a5d0-4d9c-838c-4644e42b4445-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-trbpt\" (UID: \"5768c384-a5d0-4d9c-838c-4644e42b4445\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:54:56.030411 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:56.030300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8sd4\" (UniqueName: \"kubernetes.io/projected/5768c384-a5d0-4d9c-838c-4644e42b4445-kube-api-access-x8sd4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-trbpt\" (UID: \"5768c384-a5d0-4d9c-838c-4644e42b4445\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:54:56.032567 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:56.032539    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5768c384-a5d0-4d9c-838c-4644e42b4445-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-trbpt\" (UID: \"5768c384-a5d0-4d9c-838c-4644e42b4445\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:54:56.053159 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:56.053134    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8sd4\" (UniqueName: \"kubernetes.io/projected/5768c384-a5d0-4d9c-838c-4644e42b4445-kube-api-access-x8sd4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-trbpt\" (UID: \"5768c384-a5d0-4d9c-838c-4644e42b4445\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:54:56.193395 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:56.193367    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:54:56.472628 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:56.472605    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"]
Sep 16 18:54:56.474957 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:54:56.474932    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5768c384_a5d0_4d9c_838c_4644e42b4445.slice/crio-77ce971c3f850ec7885b915d235a2b27ed725af6f69a91ae6a90c4135c17c746 WatchSource:0}: Error finding container 77ce971c3f850ec7885b915d235a2b27ed725af6f69a91ae6a90c4135c17c746: Status 404 returned error can't find the container with id 77ce971c3f850ec7885b915d235a2b27ed725af6f69a91ae6a90c4135c17c746
Sep 16 18:54:56.476649 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:56.476632    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:54:56.865644 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:54:56.865562    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt" event={"ID":"5768c384-a5d0-4d9c-838c-4644e42b4445","Type":"ContainerStarted","Data":"77ce971c3f850ec7885b915d235a2b27ed725af6f69a91ae6a90c4135c17c746"}
Sep 16 18:55:00.894193 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:00.894149    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt" event={"ID":"5768c384-a5d0-4d9c-838c-4644e42b4445","Type":"ContainerStarted","Data":"4d02421ea32ec2bb3b0df12d0528b0e98da34effa03514183e3ace95f3391f2d"}
Sep 16 18:55:00.894580 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:00.894334    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:55:00.914165 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:00.914115    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt" podStartSLOduration=2.43114607 podStartE2EDuration="5.914099246s" totalImagesPullingTime="3.482953176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:00.913462014 +0000 UTC m=+651.107325687" watchObservedRunningTime="2026-09-16 18:55:00.914099246 +0000 UTC m=+651.107962918"
Sep 16 18:55:01.238320 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.238280    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-wcns8"]
Sep 16 18:55:01.242211 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.242190    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:01.246444 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.246420    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:55:01.246676 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.246655    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7jlxm\""
Sep 16 18:55:01.246790 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.246762    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:55:01.268596 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.268568    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-wcns8"]
Sep 16 18:55:01.273971 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.273941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/30e0859e-d6c1-4805-95b4-119935d19094-certificates\") pod \"keda-admission-6c4d864b86-wcns8\" (UID: \"30e0859e-d6c1-4805-95b4-119935d19094\") " pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:01.274095 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.273986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57zc\" (UniqueName: \"kubernetes.io/projected/30e0859e-d6c1-4805-95b4-119935d19094-kube-api-access-x57zc\") pod \"keda-admission-6c4d864b86-wcns8\" (UID: \"30e0859e-d6c1-4805-95b4-119935d19094\") " pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:01.375103 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.375058    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/30e0859e-d6c1-4805-95b4-119935d19094-certificates\") pod \"keda-admission-6c4d864b86-wcns8\" (UID: \"30e0859e-d6c1-4805-95b4-119935d19094\") " pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:01.375274 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.375109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57zc\" (UniqueName: \"kubernetes.io/projected/30e0859e-d6c1-4805-95b4-119935d19094-kube-api-access-x57zc\") pod \"keda-admission-6c4d864b86-wcns8\" (UID: \"30e0859e-d6c1-4805-95b4-119935d19094\") " pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:01.375274 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:55:01.375204    2576 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 16 18:55:01.375274 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:55:01.375239    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-wcns8: secret "keda-admission-webhooks-certs" not found
Sep 16 18:55:01.375402 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:55:01.375293    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30e0859e-d6c1-4805-95b4-119935d19094-certificates podName:30e0859e-d6c1-4805-95b4-119935d19094 nodeName:}" failed. No retries permitted until 2026-09-16 18:55:01.875277123 +0000 UTC m=+652.069140777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/30e0859e-d6c1-4805-95b4-119935d19094-certificates") pod "keda-admission-6c4d864b86-wcns8" (UID: "30e0859e-d6c1-4805-95b4-119935d19094") : secret "keda-admission-webhooks-certs" not found
Sep 16 18:55:01.424888 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.424856    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57zc\" (UniqueName: \"kubernetes.io/projected/30e0859e-d6c1-4805-95b4-119935d19094-kube-api-access-x57zc\") pod \"keda-admission-6c4d864b86-wcns8\" (UID: \"30e0859e-d6c1-4805-95b4-119935d19094\") " pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:01.881435 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.881400    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/30e0859e-d6c1-4805-95b4-119935d19094-certificates\") pod \"keda-admission-6c4d864b86-wcns8\" (UID: \"30e0859e-d6c1-4805-95b4-119935d19094\") " pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:01.883809 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:01.883788    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/30e0859e-d6c1-4805-95b4-119935d19094-certificates\") pod \"keda-admission-6c4d864b86-wcns8\" (UID: \"30e0859e-d6c1-4805-95b4-119935d19094\") " pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:02.152750 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:02.152666    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:02.427408 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:02.427368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-wcns8"]
Sep 16 18:55:02.430551 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:55:02.430527    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e0859e_d6c1_4805_95b4_119935d19094.slice/crio-87b656e877acb7050ca6beec0c5f5b56d184c3ebbbd75fd95906c2b608a53646 WatchSource:0}: Error finding container 87b656e877acb7050ca6beec0c5f5b56d184c3ebbbd75fd95906c2b608a53646: Status 404 returned error can't find the container with id 87b656e877acb7050ca6beec0c5f5b56d184c3ebbbd75fd95906c2b608a53646
Sep 16 18:55:02.905068 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:02.904988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-wcns8" event={"ID":"30e0859e-d6c1-4805-95b4-119935d19094","Type":"ContainerStarted","Data":"87b656e877acb7050ca6beec0c5f5b56d184c3ebbbd75fd95906c2b608a53646"}
Sep 16 18:55:05.920344 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:05.920316    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-wcns8" event={"ID":"30e0859e-d6c1-4805-95b4-119935d19094","Type":"ContainerStarted","Data":"a956311898ddb9b45cf8fa32aeabb141182658e25cfe297fd24d17c1ee7cd0a5"}
Sep 16 18:55:05.920720 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:05.920426    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:55:05.934905 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:05.934861    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-wcns8" podStartSLOduration=2.074204896 podStartE2EDuration="4.934822658s" totalImagesPullingTime="2.860617762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:05.93427846 +0000 UTC m=+656.128142134" watchObservedRunningTime="2026-09-16 18:55:05.934822658 +0000 UTC m=+656.128686330"
Sep 16 18:55:21.901005 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:21.900975    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-trbpt"
Sep 16 18:55:26.927538 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:55:26.927505    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-wcns8"
Sep 16 18:56:17.960123 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:17.960088    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"]
Sep 16 18:56:17.963391 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:17.963375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:17.966056 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:17.966032    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:56:17.966803 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:17.966778    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-qk9hg\""
Sep 16 18:56:17.966942 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:17.966802    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:56:17.966942 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:17.966815    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:56:17.975203 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:17.975183    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"]
Sep 16 18:56:18.094547 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:18.094514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflkt\" (UniqueName: \"kubernetes.io/projected/5a0fda36-e8fe-44c8-8865-bde5526ca6fb-kube-api-access-nflkt\") pod \"llmisvc-controller-manager-645f9c8fd6-bn2h6\" (UID: \"5a0fda36-e8fe-44c8-8865-bde5526ca6fb\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:18.094722 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:18.094575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a0fda36-e8fe-44c8-8865-bde5526ca6fb-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bn2h6\" (UID: \"5a0fda36-e8fe-44c8-8865-bde5526ca6fb\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:18.195776 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:18.195736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflkt\" (UniqueName: \"kubernetes.io/projected/5a0fda36-e8fe-44c8-8865-bde5526ca6fb-kube-api-access-nflkt\") pod \"llmisvc-controller-manager-645f9c8fd6-bn2h6\" (UID: \"5a0fda36-e8fe-44c8-8865-bde5526ca6fb\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:18.195988 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:18.195898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a0fda36-e8fe-44c8-8865-bde5526ca6fb-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bn2h6\" (UID: \"5a0fda36-e8fe-44c8-8865-bde5526ca6fb\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:18.195988 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:56:18.195955    2576 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 18:56:18.196103 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:56:18.196016    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0fda36-e8fe-44c8-8865-bde5526ca6fb-cert podName:5a0fda36-e8fe-44c8-8865-bde5526ca6fb nodeName:}" failed. No retries permitted until 2026-09-16 18:56:18.695997787 +0000 UTC m=+728.889861442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a0fda36-e8fe-44c8-8865-bde5526ca6fb-cert") pod "llmisvc-controller-manager-645f9c8fd6-bn2h6" (UID: "5a0fda36-e8fe-44c8-8865-bde5526ca6fb") : secret "llmisvc-webhook-server-cert" not found
Sep 16 18:56:18.204618 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:18.204594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflkt\" (UniqueName: \"kubernetes.io/projected/5a0fda36-e8fe-44c8-8865-bde5526ca6fb-kube-api-access-nflkt\") pod \"llmisvc-controller-manager-645f9c8fd6-bn2h6\" (UID: \"5a0fda36-e8fe-44c8-8865-bde5526ca6fb\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:18.700752 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:18.700724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a0fda36-e8fe-44c8-8865-bde5526ca6fb-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bn2h6\" (UID: \"5a0fda36-e8fe-44c8-8865-bde5526ca6fb\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:18.703099 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:18.703069    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a0fda36-e8fe-44c8-8865-bde5526ca6fb-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bn2h6\" (UID: \"5a0fda36-e8fe-44c8-8865-bde5526ca6fb\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:18.874422 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:18.874386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:19.143051 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:19.143025    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"]
Sep 16 18:56:19.145717 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:56:19.145691    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5a0fda36_e8fe_44c8_8865_bde5526ca6fb.slice/crio-2c0307467010b477209d2f854d35194c728f58cbdde8c9788aff3b80128a92cc WatchSource:0}: Error finding container 2c0307467010b477209d2f854d35194c728f58cbdde8c9788aff3b80128a92cc: Status 404 returned error can't find the container with id 2c0307467010b477209d2f854d35194c728f58cbdde8c9788aff3b80128a92cc
Sep 16 18:56:19.243219 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:19.243183    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6" event={"ID":"5a0fda36-e8fe-44c8-8865-bde5526ca6fb","Type":"ContainerStarted","Data":"2c0307467010b477209d2f854d35194c728f58cbdde8c9788aff3b80128a92cc"}
Sep 16 18:56:22.259902 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:22.259869    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6" event={"ID":"5a0fda36-e8fe-44c8-8865-bde5526ca6fb","Type":"ContainerStarted","Data":"d35f964b34c27d036b10c5ed9cf539566b9c193454a6a798876064c4dc1f0731"}
Sep 16 18:56:22.260262 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:22.260035    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:56:22.274572 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:22.274525    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6" podStartSLOduration=2.8511828489999997 podStartE2EDuration="5.274511423s" totalImagesPullingTime="2.423328574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:22.274025647 +0000 UTC m=+732.467889320" watchObservedRunningTime="2026-09-16 18:56:22.274511423 +0000 UTC m=+732.468375134"
Sep 16 18:56:53.266921 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:56:53.266887    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bn2h6"
Sep 16 18:57:51.733152 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.733079    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-b7tbx"]
Sep 16 18:57:51.736564 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.736549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-b7tbx"
Sep 16 18:57:51.738618 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.738593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58cq\" (UniqueName: \"kubernetes.io/projected/40eb6d96-5a5a-4878-968c-866566f99a0a-kube-api-access-d58cq\") pod \"s3-init-b7tbx\" (UID: \"40eb6d96-5a5a-4878-968c-866566f99a0a\") " pod="kserve/s3-init-b7tbx"
Sep 16 18:57:51.738735 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.738640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40eb6d96-5a5a-4878-968c-866566f99a0a-model-cache\") pod \"s3-init-b7tbx\" (UID: \"40eb6d96-5a5a-4878-968c-866566f99a0a\") " pod="kserve/s3-init-b7tbx"
Sep 16 18:57:51.738847 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.738823    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:57:51.738891 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.738826    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:57:51.738891 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.738826    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wb9h8\""
Sep 16 18:57:51.742974 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.742948    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-b7tbx"]
Sep 16 18:57:51.839311 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.839286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58cq\" (UniqueName: \"kubernetes.io/projected/40eb6d96-5a5a-4878-968c-866566f99a0a-kube-api-access-d58cq\") pod \"s3-init-b7tbx\" (UID: \"40eb6d96-5a5a-4878-968c-866566f99a0a\") " pod="kserve/s3-init-b7tbx"
Sep 16 18:57:51.839429 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.839327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40eb6d96-5a5a-4878-968c-866566f99a0a-model-cache\") pod \"s3-init-b7tbx\" (UID: \"40eb6d96-5a5a-4878-968c-866566f99a0a\") " pod="kserve/s3-init-b7tbx"
Sep 16 18:57:51.839640 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.839621    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40eb6d96-5a5a-4878-968c-866566f99a0a-model-cache\") pod \"s3-init-b7tbx\" (UID: \"40eb6d96-5a5a-4878-968c-866566f99a0a\") " pod="kserve/s3-init-b7tbx"
Sep 16 18:57:51.848029 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:51.848009    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58cq\" (UniqueName: \"kubernetes.io/projected/40eb6d96-5a5a-4878-968c-866566f99a0a-kube-api-access-d58cq\") pod \"s3-init-b7tbx\" (UID: \"40eb6d96-5a5a-4878-968c-866566f99a0a\") " pod="kserve/s3-init-b7tbx"
Sep 16 18:57:52.057535 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:52.057449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-b7tbx"
Sep 16 18:57:52.326423 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:52.326398    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-b7tbx"]
Sep 16 18:57:52.649898 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:52.649804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b7tbx" event={"ID":"40eb6d96-5a5a-4878-968c-866566f99a0a","Type":"ContainerStarted","Data":"e17d26e580a5400fe9e4fca2b146e1de4a7facd2e767277b96e07c7c93fa1457"}
Sep 16 18:57:55.666351 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:55.666314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b7tbx" event={"ID":"40eb6d96-5a5a-4878-968c-866566f99a0a","Type":"ContainerStarted","Data":"1dd979ab92ed54197f7903f29f3c696357164b17c88fe206df7a070394989241"}
Sep 16 18:57:57.677898 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:57.677865    2576 generic.go:356] "Generic (PLEG): container finished" podID="40eb6d96-5a5a-4878-968c-866566f99a0a" containerID="1dd979ab92ed54197f7903f29f3c696357164b17c88fe206df7a070394989241" exitCode=0
Sep 16 18:57:57.678276 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:57:57.677904    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b7tbx" event={"ID":"40eb6d96-5a5a-4878-968c-866566f99a0a","Type":"ContainerDied","Data":"1dd979ab92ed54197f7903f29f3c696357164b17c88fe206df7a070394989241"}
Sep 16 18:58:02.704999 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:58:02.704961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b7tbx" event={"ID":"40eb6d96-5a5a-4878-968c-866566f99a0a","Type":"ContainerStarted","Data":"43a5a081b6fec5e296f97ed7390a5a4da730b8e5281bfef6e460c41e7715a5ba"}
Sep 16 18:58:02.720295 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:58:02.720249    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-b7tbx" podStartSLOduration=2.2570797479999998 podStartE2EDuration="11.720232148s" totalImagesPullingTime="7.52290508s" totalInitContainerRuntime="1.94024732s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:02.719997628 +0000 UTC m=+832.913861301" watchObservedRunningTime="2026-09-16 18:58:02.720232148 +0000 UTC m=+832.914095822"
Sep 16 18:59:06.996135 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:06.996101    2576 generic.go:356] "Generic (PLEG): container finished" podID="40eb6d96-5a5a-4878-968c-866566f99a0a" containerID="43a5a081b6fec5e296f97ed7390a5a4da730b8e5281bfef6e460c41e7715a5ba" exitCode=0
Sep 16 18:59:06.996507 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:06.996172    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b7tbx" event={"ID":"40eb6d96-5a5a-4878-968c-866566f99a0a","Type":"ContainerDied","Data":"43a5a081b6fec5e296f97ed7390a5a4da730b8e5281bfef6e460c41e7715a5ba"}
Sep 16 18:59:08.124306 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:08.124275    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-b7tbx"
Sep 16 18:59:08.182037 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:08.182009    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d58cq\" (UniqueName: \"kubernetes.io/projected/40eb6d96-5a5a-4878-968c-866566f99a0a-kube-api-access-d58cq\") pod \"40eb6d96-5a5a-4878-968c-866566f99a0a\" (UID: \"40eb6d96-5a5a-4878-968c-866566f99a0a\") "
Sep 16 18:59:08.182196 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:08.182075    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40eb6d96-5a5a-4878-968c-866566f99a0a-model-cache\") pod \"40eb6d96-5a5a-4878-968c-866566f99a0a\" (UID: \"40eb6d96-5a5a-4878-968c-866566f99a0a\") "
Sep 16 18:59:08.184140 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:08.184116    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40eb6d96-5a5a-4878-968c-866566f99a0a-kube-api-access-d58cq" (OuterVolumeSpecName: "kube-api-access-d58cq") pod "40eb6d96-5a5a-4878-968c-866566f99a0a" (UID: "40eb6d96-5a5a-4878-968c-866566f99a0a"). InnerVolumeSpecName "kube-api-access-d58cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:08.204166 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:08.204131    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eb6d96-5a5a-4878-968c-866566f99a0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "40eb6d96-5a5a-4878-968c-866566f99a0a" (UID: "40eb6d96-5a5a-4878-968c-866566f99a0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:08.283512 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:08.283440    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58cq\" (UniqueName: \"kubernetes.io/projected/40eb6d96-5a5a-4878-968c-866566f99a0a-kube-api-access-d58cq\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:59:08.283512 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:08.283467    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40eb6d96-5a5a-4878-968c-866566f99a0a-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 18:59:09.006104 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:09.006077    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-b7tbx"
Sep 16 18:59:09.006269 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:09.006082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b7tbx" event={"ID":"40eb6d96-5a5a-4878-968c-866566f99a0a","Type":"ContainerDied","Data":"e17d26e580a5400fe9e4fca2b146e1de4a7facd2e767277b96e07c7c93fa1457"}
Sep 16 18:59:09.006269 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:09.006195    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e17d26e580a5400fe9e4fca2b146e1de4a7facd2e767277b96e07c7c93fa1457"
Sep 16 18:59:10.318900 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:10.318872    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:59:10.319301 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:10.319196    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 18:59:20.972356 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.972322    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"]
Sep 16 18:59:20.972813 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.972682    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40eb6d96-5a5a-4878-968c-866566f99a0a" containerName="s3-init"
Sep 16 18:59:20.972813 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.972693    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40eb6d96-5a5a-4878-968c-866566f99a0a" containerName="s3-init"
Sep 16 18:59:20.972813 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.972704    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40eb6d96-5a5a-4878-968c-866566f99a0a" containerName="download-hf-model"
Sep 16 18:59:20.972813 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.972710    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40eb6d96-5a5a-4878-968c-866566f99a0a" containerName="download-hf-model"
Sep 16 18:59:20.972813 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.972766    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="40eb6d96-5a5a-4878-968c-866566f99a0a" containerName="s3-init"
Sep 16 18:59:20.976068 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.976052    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:20.978200 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.978177    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Sep 16 18:59:20.978991 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.978972    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:59:20.979235 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.979213    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:59:20.979339 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.979215    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dgjbs\""
Sep 16 18:59:20.979339 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.979215    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:59:20.979581 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.979562    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:59:20.979683 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.979662    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Sep 16 18:59:20.993505 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:20.993483    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"]
Sep 16 18:59:21.092130 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.092087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b6bm\" (UniqueName: \"kubernetes.io/projected/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-api-access-5b6bm\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.092314 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.092166    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.092314 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.092219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.092314 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.092282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/488c6d0b-019c-4317-9179-ce5a400bba8f-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.092314 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.092309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/488c6d0b-019c-4317-9179-ce5a400bba8f-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.145187 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.145148    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"]
Sep 16 18:59:21.149246 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.149224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.151722 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.151692    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-658c9-kube-rbac-proxy-sar-config\""
Sep 16 18:59:21.151877 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.151861    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-658c9-predictor-serving-cert\""
Sep 16 18:59:21.157717 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.157695    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"]
Sep 16 18:59:21.193476 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.193440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.193619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.193506    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.193619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.193551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/488c6d0b-019c-4317-9179-ce5a400bba8f-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.193619 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.193579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/488c6d0b-019c-4317-9179-ce5a400bba8f-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.193787 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.193618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b6bm\" (UniqueName: \"kubernetes.io/projected/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-api-access-5b6bm\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.194234 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:59:21.194212    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 16 18:59:21.194330 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:59:21.194300    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/488c6d0b-019c-4317-9179-ce5a400bba8f-proxy-tls podName:488c6d0b-019c-4317-9179-ce5a400bba8f nodeName:}" failed. No retries permitted until 2026-09-16 18:59:21.694281928 +0000 UTC m=+911.888145591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/488c6d0b-019c-4317-9179-ce5a400bba8f-proxy-tls") pod "isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" (UID: "488c6d0b-019c-4317-9179-ce5a400bba8f") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 16 18:59:21.194406 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.194229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/488c6d0b-019c-4317-9179-ce5a400bba8f-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.195054 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.195021    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.195054 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.195032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.203650 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.203629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b6bm\" (UniqueName: \"kubernetes.io/projected/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-api-access-5b6bm\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.274736 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.274653    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"]
Sep 16 18:59:21.278611 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.278589    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.280874 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.280852    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 16 18:59:21.280994 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.280974    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 16 18:59:21.288490 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.288468    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"]
Sep 16 18:59:21.294925 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.294901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-proxy-tls\") pod \"error-404-isvc-658c9-predictor-7c978db87d-d4q87\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") " pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.295050 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.294976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8sd\" (UniqueName: \"kubernetes.io/projected/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-api-access-4t8sd\") pod \"error-404-isvc-658c9-predictor-7c978db87d-d4q87\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") " pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.295116 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.295059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-658c9-predictor-7c978db87d-d4q87\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") " pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.395655 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.395617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8sd\" (UniqueName: \"kubernetes.io/projected/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-api-access-4t8sd\") pod \"error-404-isvc-658c9-predictor-7c978db87d-d4q87\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") " pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.395655 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.395656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348afb3f-512c-4d84-99af-d4480160c6c8-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.395929 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.395706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-658c9-predictor-7c978db87d-d4q87\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") " pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.395929 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.395736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgl9\" (UniqueName: \"kubernetes.io/projected/348afb3f-512c-4d84-99af-d4480160c6c8-kube-api-access-4qgl9\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.395929 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.395828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-proxy-tls\") pod \"error-404-isvc-658c9-predictor-7c978db87d-d4q87\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") " pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.395929 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.395881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.396137 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.395932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348afb3f-512c-4d84-99af-d4480160c6c8-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.396137 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.395959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.396457 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.396440    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-658c9-predictor-7c978db87d-d4q87\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") " pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.398520 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.398493    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-proxy-tls\") pod \"error-404-isvc-658c9-predictor-7c978db87d-d4q87\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") " pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.403985 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.403957    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8sd\" (UniqueName: \"kubernetes.io/projected/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-api-access-4t8sd\") pod \"error-404-isvc-658c9-predictor-7c978db87d-d4q87\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") " pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.461595 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.461563    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:21.497130 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.497095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgl9\" (UniqueName: \"kubernetes.io/projected/348afb3f-512c-4d84-99af-d4480160c6c8-kube-api-access-4qgl9\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.497271 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.497183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.497271 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.497227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348afb3f-512c-4d84-99af-d4480160c6c8-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.497271 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.497250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.497425 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.497302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348afb3f-512c-4d84-99af-d4480160c6c8-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.497475 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:59:21.497438    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 16 18:59:21.497510 ip-10-0-130-134 kubenswrapper[2576]: E0916 18:59:21.497504    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348afb3f-512c-4d84-99af-d4480160c6c8-proxy-tls podName:348afb3f-512c-4d84-99af-d4480160c6c8 nodeName:}" failed. No retries permitted until 2026-09-16 18:59:21.997484972 +0000 UTC m=+912.191348629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/348afb3f-512c-4d84-99af-d4480160c6c8-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-2xnps" (UID: "348afb3f-512c-4d84-99af-d4480160c6c8") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 16 18:59:21.497807 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.497783    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348afb3f-512c-4d84-99af-d4480160c6c8-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.498111 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.498084    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.498339 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.498319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.505600 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.505576    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgl9\" (UniqueName: \"kubernetes.io/projected/348afb3f-512c-4d84-99af-d4480160c6c8-kube-api-access-4qgl9\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:21.699395 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.699363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/488c6d0b-019c-4317-9179-ce5a400bba8f-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.701722 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.701701    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/488c6d0b-019c-4317-9179-ce5a400bba8f-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:21.734436 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.734411    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"]
Sep 16 18:59:21.736295 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:59:21.736271    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ded47fc_0c9c_4f99_a2d2_67528c5d81ed.slice/crio-4afae867ec8abac64d2bbeb63b6239f49e1c65218e1bc699221404d4bd198bb9 WatchSource:0}: Error finding container 4afae867ec8abac64d2bbeb63b6239f49e1c65218e1bc699221404d4bd198bb9: Status 404 returned error can't find the container with id 4afae867ec8abac64d2bbeb63b6239f49e1c65218e1bc699221404d4bd198bb9
Sep 16 18:59:21.888798 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:21.888754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:22.001416 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:22.001348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348afb3f-512c-4d84-99af-d4480160c6c8-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:22.003684 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:22.003659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348afb3f-512c-4d84-99af-d4480160c6c8-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2xnps\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:22.066409 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:22.066383    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" event={"ID":"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed","Type":"ContainerStarted","Data":"4afae867ec8abac64d2bbeb63b6239f49e1c65218e1bc699221404d4bd198bb9"}
Sep 16 18:59:22.163333 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:22.163301    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"]
Sep 16 18:59:22.166079 ip-10-0-130-134 kubenswrapper[2576]: W0916 18:59:22.166055    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488c6d0b_019c_4317_9179_ce5a400bba8f.slice/crio-7f68f673cb561f3381b1b5154478f31c0d607c3526eec2988e7871a04865e051 WatchSource:0}: Error finding container 7f68f673cb561f3381b1b5154478f31c0d607c3526eec2988e7871a04865e051: Status 404 returned error can't find the container with id 7f68f673cb561f3381b1b5154478f31c0d607c3526eec2988e7871a04865e051
Sep 16 18:59:22.191340 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:22.191316    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:22.471998 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:22.471963    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"]
Sep 16 18:59:23.080311 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:23.079414    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" event={"ID":"348afb3f-512c-4d84-99af-d4480160c6c8","Type":"ContainerStarted","Data":"97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a"}
Sep 16 18:59:23.080311 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:23.079493    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" event={"ID":"348afb3f-512c-4d84-99af-d4480160c6c8","Type":"ContainerStarted","Data":"5b21fbcfbcdf9017932490ea9635929becb8e7f1df4983e5a507f2d7d10eeac3"}
Sep 16 18:59:23.085457 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:23.084943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" event={"ID":"488c6d0b-019c-4317-9179-ce5a400bba8f","Type":"ContainerStarted","Data":"de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843"}
Sep 16 18:59:23.085457 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:23.085013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" event={"ID":"488c6d0b-019c-4317-9179-ce5a400bba8f","Type":"ContainerStarted","Data":"7f68f673cb561f3381b1b5154478f31c0d607c3526eec2988e7871a04865e051"}
Sep 16 18:59:27.114953 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:27.114864    2576 generic.go:356] "Generic (PLEG): container finished" podID="348afb3f-512c-4d84-99af-d4480160c6c8" containerID="97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a" exitCode=0
Sep 16 18:59:27.115388 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:27.114967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" event={"ID":"348afb3f-512c-4d84-99af-d4480160c6c8","Type":"ContainerDied","Data":"97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a"}
Sep 16 18:59:27.116961 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:27.116933    2576 generic.go:356] "Generic (PLEG): container finished" podID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerID="de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843" exitCode=0
Sep 16 18:59:27.117096 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:27.116971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" event={"ID":"488c6d0b-019c-4317-9179-ce5a400bba8f","Type":"ContainerDied","Data":"de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843"}
Sep 16 18:59:52.278862 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:52.278802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" event={"ID":"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed","Type":"ContainerStarted","Data":"cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f"}
Sep 16 18:59:52.282617 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:52.282586    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" event={"ID":"488c6d0b-019c-4317-9179-ce5a400bba8f","Type":"ContainerStarted","Data":"d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493"}
Sep 16 18:59:53.292754 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:53.292637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" event={"ID":"348afb3f-512c-4d84-99af-d4480160c6c8","Type":"ContainerStarted","Data":"fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224"}
Sep 16 18:59:55.305080 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.305046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" event={"ID":"488c6d0b-019c-4317-9179-ce5a400bba8f","Type":"ContainerStarted","Data":"689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c"}
Sep 16 18:59:55.305541 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.305118    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:55.305541 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.305176    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 18:59:55.306733 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.306706    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:59:55.307197 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.307167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" event={"ID":"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed","Type":"ContainerStarted","Data":"9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b"}
Sep 16 18:59:55.307317 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.307221    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:55.309453 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.309427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" event={"ID":"348afb3f-512c-4d84-99af-d4480160c6c8","Type":"ContainerStarted","Data":"5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0"}
Sep 16 18:59:55.309593 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.309583    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:55.324900 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.324858    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podStartSLOduration=4.477965124 podStartE2EDuration="35.324831493s" totalImagesPullingTime="26.805376145s" totalInitContainerRuntime="4.041490224s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:55.322561962 +0000 UTC m=+945.516425642" watchObservedRunningTime="2026-09-16 18:59:55.324831493 +0000 UTC m=+945.518695165"
Sep 16 18:59:55.339559 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.339519    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podStartSLOduration=10.672106822 podStartE2EDuration="34.339506164s" totalImagesPullingTime="23.667399342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:55.339077685 +0000 UTC m=+945.532941359" watchObservedRunningTime="2026-09-16 18:59:55.339506164 +0000 UTC m=+945.533369835"
Sep 16 18:59:55.357404 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:55.357351    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podStartSLOduration=2.961994031 podStartE2EDuration="34.357338151s" totalImagesPullingTime="27.326679453s" totalInitContainerRuntime="4.068664667s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:55.356106783 +0000 UTC m=+945.549970456" watchObservedRunningTime="2026-09-16 18:59:55.357338151 +0000 UTC m=+945.551201822"
Sep 16 18:59:56.315002 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:56.314961    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:59:56.317379 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:56.315124    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 18:59:56.317379 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:56.315196    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 18:59:56.317379 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:56.315777    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:56.317379 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:56.315793    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:59:57.319233 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:57.319176    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:59:57.319233 ip-10-0-130-134 kubenswrapper[2576]: I0916 18:59:57.319218    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:01.319240 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:01.319212    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 19:00:01.319738 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:01.319709    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:00:02.324026 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:02.323999    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 19:00:02.324431 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:02.324393    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 19:00:02.324500 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:02.324447    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:02.324861 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:02.324819    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:00:11.319712 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:11.319667    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:00:12.324485 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:12.324443    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:12.324891 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:12.324860    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:00:21.320197 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:21.320158    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:00:22.325240 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:22.325200    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:00:22.325674 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:22.325200    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:31.320326 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:31.320289    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:00:32.325207 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:32.325123    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:00:32.325578 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:32.325123    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:41.319692 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:41.319654    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:00:42.325095 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:42.325055    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:42.325628 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:42.325610    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 19:00:51.319710 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:51.319671    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:00:52.324771 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:00:52.324730    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:01:01.154495 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.154453    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"]
Sep 16 19:01:01.154872 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.154740    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container" containerID="cri-o://cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f" gracePeriod=30
Sep 16 19:01:01.154941 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.154831    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kube-rbac-proxy" containerID="cri-o://9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b" gracePeriod=30
Sep 16 19:01:01.319996 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.319963    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:01:01.413833 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.413802    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"]
Sep 16 19:01:01.417425 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.417408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.419772 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.419746    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-93efe-predictor-serving-cert\""
Sep 16 19:01:01.420067 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.420048    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-93efe-kube-rbac-proxy-sar-config\""
Sep 16 19:01:01.439185 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.439162    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"]
Sep 16 19:01:01.493708 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.493673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcwf\" (UniqueName: \"kubernetes.io/projected/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-api-access-zmcwf\") pod \"error-404-isvc-93efe-predictor-6486b7df5b-vl2jt\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") " pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.493904 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.493729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-93efe-predictor-6486b7df5b-vl2jt\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") " pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.493904 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.493799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0b0141-408d-4222-b40f-6ca6c655f912-proxy-tls\") pod \"error-404-isvc-93efe-predictor-6486b7df5b-vl2jt\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") " pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.594626 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.594598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-93efe-predictor-6486b7df5b-vl2jt\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") " pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.594796 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.594652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0b0141-408d-4222-b40f-6ca6c655f912-proxy-tls\") pod \"error-404-isvc-93efe-predictor-6486b7df5b-vl2jt\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") " pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.594796 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.594679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcwf\" (UniqueName: \"kubernetes.io/projected/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-api-access-zmcwf\") pod \"error-404-isvc-93efe-predictor-6486b7df5b-vl2jt\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") " pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.595293 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.595271    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-93efe-predictor-6486b7df5b-vl2jt\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") " pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.597119 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.597096    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0b0141-408d-4222-b40f-6ca6c655f912-proxy-tls\") pod \"error-404-isvc-93efe-predictor-6486b7df5b-vl2jt\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") " pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.604170 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.604143    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcwf\" (UniqueName: \"kubernetes.io/projected/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-api-access-zmcwf\") pod \"error-404-isvc-93efe-predictor-6486b7df5b-vl2jt\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") " pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.625426 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.625402    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerID="9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b" exitCode=2
Sep 16 19:01:01.625506 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.625439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" event={"ID":"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed","Type":"ContainerDied","Data":"9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b"}
Sep 16 19:01:01.728251 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.728166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:01.999918 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:01.999890    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"]
Sep 16 19:01:02.002358 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:01:02.002332    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0b0141_408d_4222_b40f_6ca6c655f912.slice/crio-d53f711fa83b2e64c70eeb67840e24e0772340c686f27415240364a504605bef WatchSource:0}: Error finding container d53f711fa83b2e64c70eeb67840e24e0772340c686f27415240364a504605bef: Status 404 returned error can't find the container with id d53f711fa83b2e64c70eeb67840e24e0772340c686f27415240364a504605bef
Sep 16 19:01:02.320321 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.320228    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 19:01:02.325120 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.325095    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 19:01:02.325343 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.325321    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:01:02.633002 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.632905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" event={"ID":"4f0b0141-408d-4222-b40f-6ca6c655f912","Type":"ContainerStarted","Data":"a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c"}
Sep 16 19:01:02.633002 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.632966    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" event={"ID":"4f0b0141-408d-4222-b40f-6ca6c655f912","Type":"ContainerStarted","Data":"09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee"}
Sep 16 19:01:02.633002 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.632986    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" event={"ID":"4f0b0141-408d-4222-b40f-6ca6c655f912","Type":"ContainerStarted","Data":"d53f711fa83b2e64c70eeb67840e24e0772340c686f27415240364a504605bef"}
Sep 16 19:01:02.633270 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.633009    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:02.633270 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.633051    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:02.634658 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.634635    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:02.655391 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:02.655326    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podStartSLOduration=1.655309103 podStartE2EDuration="1.655309103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:02.653344874 +0000 UTC m=+1012.847208577" watchObservedRunningTime="2026-09-16 19:01:02.655309103 +0000 UTC m=+1012.849172774"
Sep 16 19:01:03.638360 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:03.638325    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:04.402661 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.402634    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 19:01:04.516443 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.516370    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-rbac-proxy-sar-config\") pod \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") "
Sep 16 19:01:04.516584 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.516462    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t8sd\" (UniqueName: \"kubernetes.io/projected/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-api-access-4t8sd\") pod \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") "
Sep 16 19:01:04.516584 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.516481    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-proxy-tls\") pod \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\" (UID: \"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed\") "
Sep 16 19:01:04.516792 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.516764    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" (UID: "1ded47fc-0c9c-4f99-a2d2-67528c5d81ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:04.518626 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.518602    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-api-access-4t8sd" (OuterVolumeSpecName: "kube-api-access-4t8sd") pod "1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" (UID: "1ded47fc-0c9c-4f99-a2d2-67528c5d81ed"). InnerVolumeSpecName "kube-api-access-4t8sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:04.518706 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.518604    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" (UID: "1ded47fc-0c9c-4f99-a2d2-67528c5d81ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:04.617360 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.617328    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:04.617360 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.617355    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t8sd\" (UniqueName: \"kubernetes.io/projected/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-kube-api-access-4t8sd\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:04.617360 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.617365    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:04.643771 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.643736    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerID="cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f" exitCode=0
Sep 16 19:01:04.644167 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.643793    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"
Sep 16 19:01:04.644167 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.643820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" event={"ID":"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed","Type":"ContainerDied","Data":"cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f"}
Sep 16 19:01:04.644167 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.643895    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87" event={"ID":"1ded47fc-0c9c-4f99-a2d2-67528c5d81ed","Type":"ContainerDied","Data":"4afae867ec8abac64d2bbeb63b6239f49e1c65218e1bc699221404d4bd198bb9"}
Sep 16 19:01:04.644167 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.643925    2576 scope.go:117] "RemoveContainer" containerID="9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b"
Sep 16 19:01:04.653142 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.652872    2576 scope.go:117] "RemoveContainer" containerID="cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f"
Sep 16 19:01:04.661284 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.661267    2576 scope.go:117] "RemoveContainer" containerID="9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b"
Sep 16 19:01:04.661522 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:04.661505    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b\": container with ID starting with 9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b not found: ID does not exist" containerID="9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b"
Sep 16 19:01:04.661573 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.661530    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b"} err="failed to get container status \"9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b\": rpc error: code = NotFound desc = could not find container \"9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b\": container with ID starting with 9a8142963ec22ddf001383c4a86980fc24f59669566dcdd2cf8d1cd92891f98b not found: ID does not exist"
Sep 16 19:01:04.661573 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.661546    2576 scope.go:117] "RemoveContainer" containerID="cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f"
Sep 16 19:01:04.661808 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:04.661788    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f\": container with ID starting with cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f not found: ID does not exist" containerID="cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f"
Sep 16 19:01:04.661949 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.661814    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f"} err="failed to get container status \"cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f\": rpc error: code = NotFound desc = could not find container \"cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f\": container with ID starting with cfb4892a07445f7dbac712e8e5125909845e331725d3bc2b4a8be7dd14c2cc4f not found: ID does not exist"
Sep 16 19:01:04.674647 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.674629    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"]
Sep 16 19:01:04.678354 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:04.678335    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-658c9-predictor-7c978db87d-d4q87"]
Sep 16 19:01:06.335255 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:06.335223    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" path="/var/lib/kubelet/pods/1ded47fc-0c9c-4f99-a2d2-67528c5d81ed/volumes"
Sep 16 19:01:08.643332 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:08.643301    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:08.643772 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:08.643743    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:09.329926 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:09.329899    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 19:01:18.644333 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:18.644288    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:28.643917 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:28.643878    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:38.644331 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:38.644291    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:01:41.021206 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.021160    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"]
Sep 16 19:01:41.021627 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.021469    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container" containerID="cri-o://d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493" gracePeriod=30
Sep 16 19:01:41.021772 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.021556    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kube-rbac-proxy" containerID="cri-o://689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c" gracePeriod=30
Sep 16 19:01:41.061251 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.061222    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"]
Sep 16 19:01:41.061500 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.061473    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" containerID="cri-o://fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224" gracePeriod=30
Sep 16 19:01:41.061611 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.061524    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kube-rbac-proxy" containerID="cri-o://5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0" gracePeriod=30
Sep 16 19:01:41.297362 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.297285    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"]
Sep 16 19:01:41.297877 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.297861    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kube-rbac-proxy"
Sep 16 19:01:41.297931 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.297879    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kube-rbac-proxy"
Sep 16 19:01:41.297931 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.297904    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container"
Sep 16 19:01:41.297931 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.297913    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container"
Sep 16 19:01:41.298021 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.298009    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kube-rbac-proxy"
Sep 16 19:01:41.298054 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.298024    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ded47fc-0c9c-4f99-a2d2-67528c5d81ed" containerName="kserve-container"
Sep 16 19:01:41.301586 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.301568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.307675 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.307646    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cb9a3-predictor-serving-cert\""
Sep 16 19:01:41.307801 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.307686    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cb9a3-kube-rbac-proxy-sar-config\""
Sep 16 19:01:41.315102 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.315063    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 19:01:41.315974 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.315953    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"]
Sep 16 19:01:41.324801 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.324777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a563277-f8d3-4453-8449-670c1979f306-proxy-tls\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.324912 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.324809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87js4\" (UniqueName: \"kubernetes.io/projected/5a563277-f8d3-4453-8449-670c1979f306-kube-api-access-87js4\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.324960 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.324925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a563277-f8d3-4453-8449-670c1979f306-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.425599 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.425567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a563277-f8d3-4453-8449-670c1979f306-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.425779 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.425617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a563277-f8d3-4453-8449-670c1979f306-proxy-tls\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.425779 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.425634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87js4\" (UniqueName: \"kubernetes.io/projected/5a563277-f8d3-4453-8449-670c1979f306-kube-api-access-87js4\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.425779 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:41.425740    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-serving-cert: secret "error-404-isvc-cb9a3-predictor-serving-cert" not found
Sep 16 19:01:41.425998 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:41.425820    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a563277-f8d3-4453-8449-670c1979f306-proxy-tls podName:5a563277-f8d3-4453-8449-670c1979f306 nodeName:}" failed. No retries permitted until 2026-09-16 19:01:41.925799579 +0000 UTC m=+1052.119663247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a563277-f8d3-4453-8449-670c1979f306-proxy-tls") pod "error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" (UID: "5a563277-f8d3-4453-8449-670c1979f306") : secret "error-404-isvc-cb9a3-predictor-serving-cert" not found
Sep 16 19:01:41.426372 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.426350    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a563277-f8d3-4453-8449-670c1979f306-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.437065 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.437043    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87js4\" (UniqueName: \"kubernetes.io/projected/5a563277-f8d3-4453-8449-670c1979f306-kube-api-access-87js4\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.821389 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.821358    2576 generic.go:356] "Generic (PLEG): container finished" podID="348afb3f-512c-4d84-99af-d4480160c6c8" containerID="5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0" exitCode=2
Sep 16 19:01:41.821574 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.821432    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" event={"ID":"348afb3f-512c-4d84-99af-d4480160c6c8","Type":"ContainerDied","Data":"5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0"}
Sep 16 19:01:41.823745 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.823719    2576 generic.go:356] "Generic (PLEG): container finished" podID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerID="689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c" exitCode=2
Sep 16 19:01:41.823879 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.823772    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" event={"ID":"488c6d0b-019c-4317-9179-ce5a400bba8f","Type":"ContainerDied","Data":"689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c"}
Sep 16 19:01:41.929764 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.929736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a563277-f8d3-4453-8449-670c1979f306-proxy-tls\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:41.932209 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:41.932186    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a563277-f8d3-4453-8449-670c1979f306-proxy-tls\") pod \"error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") " pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:42.213681 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:42.213648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:42.319478 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:42.319441    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:01:42.324806 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:42.324776    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:01:42.495993 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:42.495964    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"]
Sep 16 19:01:42.497728 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:01:42.497702    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a563277_f8d3_4453_8449_670c1979f306.slice/crio-25379d40f5da32db781b5ad68ad19742515cdd181c51940cf3ebba839648d47d WatchSource:0}: Error finding container 25379d40f5da32db781b5ad68ad19742515cdd181c51940cf3ebba839648d47d: Status 404 returned error can't find the container with id 25379d40f5da32db781b5ad68ad19742515cdd181c51940cf3ebba839648d47d
Sep 16 19:01:42.831210 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:42.831125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" event={"ID":"5a563277-f8d3-4453-8449-670c1979f306","Type":"ContainerStarted","Data":"b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2"}
Sep 16 19:01:42.831210 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:42.831180    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" event={"ID":"5a563277-f8d3-4453-8449-670c1979f306","Type":"ContainerStarted","Data":"4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d"}
Sep 16 19:01:42.831210 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:42.831202    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" event={"ID":"5a563277-f8d3-4453-8449-670c1979f306","Type":"ContainerStarted","Data":"25379d40f5da32db781b5ad68ad19742515cdd181c51940cf3ebba839648d47d"}
Sep 16 19:01:42.831473 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:42.831251    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:42.855014 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:42.854963    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podStartSLOduration=1.85494622 podStartE2EDuration="1.85494622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:42.853496591 +0000 UTC m=+1053.047360320" watchObservedRunningTime="2026-09-16 19:01:42.85494622 +0000 UTC m=+1053.048809891"
Sep 16 19:01:43.836265 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:43.836231    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:43.837465 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:43.837438    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:01:44.812960 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.812935    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 19:01:44.843048 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.843012    2576 generic.go:356] "Generic (PLEG): container finished" podID="348afb3f-512c-4d84-99af-d4480160c6c8" containerID="fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224" exitCode=0
Sep 16 19:01:44.843348 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.843065    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"
Sep 16 19:01:44.843348 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.843099    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" event={"ID":"348afb3f-512c-4d84-99af-d4480160c6c8","Type":"ContainerDied","Data":"fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224"}
Sep 16 19:01:44.843348 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.843160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps" event={"ID":"348afb3f-512c-4d84-99af-d4480160c6c8","Type":"ContainerDied","Data":"5b21fbcfbcdf9017932490ea9635929becb8e7f1df4983e5a507f2d7d10eeac3"}
Sep 16 19:01:44.843348 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.843174    2576 scope.go:117] "RemoveContainer" containerID="5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0"
Sep 16 19:01:44.843692 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.843654    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:01:44.854619 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.854591    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348afb3f-512c-4d84-99af-d4480160c6c8-kserve-provision-location\") pod \"348afb3f-512c-4d84-99af-d4480160c6c8\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") "
Sep 16 19:01:44.854733 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.854644    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-cabundle-cert\") pod \"348afb3f-512c-4d84-99af-d4480160c6c8\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") "
Sep 16 19:01:44.854733 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.854673    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348afb3f-512c-4d84-99af-d4480160c6c8-proxy-tls\") pod \"348afb3f-512c-4d84-99af-d4480160c6c8\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") "
Sep 16 19:01:44.854733 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.854703    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-kube-rbac-proxy-sar-config\") pod \"348afb3f-512c-4d84-99af-d4480160c6c8\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") "
Sep 16 19:01:44.854911 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.854798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qgl9\" (UniqueName: \"kubernetes.io/projected/348afb3f-512c-4d84-99af-d4480160c6c8-kube-api-access-4qgl9\") pod \"348afb3f-512c-4d84-99af-d4480160c6c8\" (UID: \"348afb3f-512c-4d84-99af-d4480160c6c8\") "
Sep 16 19:01:44.855620 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.855290    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348afb3f-512c-4d84-99af-d4480160c6c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "348afb3f-512c-4d84-99af-d4480160c6c8" (UID: "348afb3f-512c-4d84-99af-d4480160c6c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:44.855620 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.855474    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "348afb3f-512c-4d84-99af-d4480160c6c8" (UID: "348afb3f-512c-4d84-99af-d4480160c6c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:44.856151 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.856112    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "348afb3f-512c-4d84-99af-d4480160c6c8" (UID: "348afb3f-512c-4d84-99af-d4480160c6c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:44.857188 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.857167    2576 scope.go:117] "RemoveContainer" containerID="fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224"
Sep 16 19:01:44.857725 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.857695    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348afb3f-512c-4d84-99af-d4480160c6c8-kube-api-access-4qgl9" (OuterVolumeSpecName: "kube-api-access-4qgl9") pod "348afb3f-512c-4d84-99af-d4480160c6c8" (UID: "348afb3f-512c-4d84-99af-d4480160c6c8"). InnerVolumeSpecName "kube-api-access-4qgl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:44.858143 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.858117    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348afb3f-512c-4d84-99af-d4480160c6c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "348afb3f-512c-4d84-99af-d4480160c6c8" (UID: "348afb3f-512c-4d84-99af-d4480160c6c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:44.874163 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.874141    2576 scope.go:117] "RemoveContainer" containerID="97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a"
Sep 16 19:01:44.881434 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.881416    2576 scope.go:117] "RemoveContainer" containerID="5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0"
Sep 16 19:01:44.881693 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:44.881672    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0\": container with ID starting with 5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0 not found: ID does not exist" containerID="5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0"
Sep 16 19:01:44.881756 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.881701    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0"} err="failed to get container status \"5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0\": rpc error: code = NotFound desc = could not find container \"5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0\": container with ID starting with 5e53d03619d571b2aabfa4a59cf85d3da45ff6ef09daf618b1fef86025e6f9e0 not found: ID does not exist"
Sep 16 19:01:44.881756 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.881718    2576 scope.go:117] "RemoveContainer" containerID="fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224"
Sep 16 19:01:44.882045 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:44.882028    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224\": container with ID starting with fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224 not found: ID does not exist" containerID="fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224"
Sep 16 19:01:44.882108 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.882050    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224"} err="failed to get container status \"fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224\": rpc error: code = NotFound desc = could not find container \"fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224\": container with ID starting with fe4be63e77d9d27d2e692bc1dc146ecf912c80394c5b468c3a709fde0c2ca224 not found: ID does not exist"
Sep 16 19:01:44.882108 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.882064    2576 scope.go:117] "RemoveContainer" containerID="97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a"
Sep 16 19:01:44.882288 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:44.882274    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a\": container with ID starting with 97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a not found: ID does not exist" containerID="97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a"
Sep 16 19:01:44.882324 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.882292    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a"} err="failed to get container status \"97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a\": rpc error: code = NotFound desc = could not find container \"97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a\": container with ID starting with 97d2261a2696d9f8d2a60b65ed751529bcf80c59ff440bfe0d5335c4724d3e2a not found: ID does not exist"
Sep 16 19:01:44.955732 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.955703    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qgl9\" (UniqueName: \"kubernetes.io/projected/348afb3f-512c-4d84-99af-d4480160c6c8-kube-api-access-4qgl9\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:44.955732 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.955733    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348afb3f-512c-4d84-99af-d4480160c6c8-kserve-provision-location\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:44.956022 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.955748    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-cabundle-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:44.956022 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.955761    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348afb3f-512c-4d84-99af-d4480160c6c8-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:44.956022 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:44.955772    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348afb3f-512c-4d84-99af-d4480160c6c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:45.170687 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:45.170656    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"]
Sep 16 19:01:45.173503 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:45.173481    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2xnps"]
Sep 16 19:01:46.335249 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.335219    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" path="/var/lib/kubelet/pods/348afb3f-512c-4d84-99af-d4480160c6c8/volumes"
Sep 16 19:01:46.370859 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.370824    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 19:01:46.469832 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.469797    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b6bm\" (UniqueName: \"kubernetes.io/projected/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-api-access-5b6bm\") pod \"488c6d0b-019c-4317-9179-ce5a400bba8f\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") "
Sep 16 19:01:46.469994 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.469873    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-cabundle-cert\") pod \"488c6d0b-019c-4317-9179-ce5a400bba8f\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") "
Sep 16 19:01:46.469994 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.469908    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/488c6d0b-019c-4317-9179-ce5a400bba8f-proxy-tls\") pod \"488c6d0b-019c-4317-9179-ce5a400bba8f\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") "
Sep 16 19:01:46.469994 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.469937    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/488c6d0b-019c-4317-9179-ce5a400bba8f-kserve-provision-location\") pod \"488c6d0b-019c-4317-9179-ce5a400bba8f\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") "
Sep 16 19:01:46.469994 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.469963    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-rbac-proxy-sar-config\") pod \"488c6d0b-019c-4317-9179-ce5a400bba8f\" (UID: \"488c6d0b-019c-4317-9179-ce5a400bba8f\") "
Sep 16 19:01:46.470316 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.470287    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "488c6d0b-019c-4317-9179-ce5a400bba8f" (UID: "488c6d0b-019c-4317-9179-ce5a400bba8f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:46.470391 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.470295    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488c6d0b-019c-4317-9179-ce5a400bba8f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "488c6d0b-019c-4317-9179-ce5a400bba8f" (UID: "488c6d0b-019c-4317-9179-ce5a400bba8f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:46.470391 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.470370    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "488c6d0b-019c-4317-9179-ce5a400bba8f" (UID: "488c6d0b-019c-4317-9179-ce5a400bba8f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:46.471828 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.471808    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-api-access-5b6bm" (OuterVolumeSpecName: "kube-api-access-5b6bm") pod "488c6d0b-019c-4317-9179-ce5a400bba8f" (UID: "488c6d0b-019c-4317-9179-ce5a400bba8f"). InnerVolumeSpecName "kube-api-access-5b6bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:46.471991 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.471975    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488c6d0b-019c-4317-9179-ce5a400bba8f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "488c6d0b-019c-4317-9179-ce5a400bba8f" (UID: "488c6d0b-019c-4317-9179-ce5a400bba8f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:46.571589 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.571555    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b6bm\" (UniqueName: \"kubernetes.io/projected/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-api-access-5b6bm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:46.571589 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.571583    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-cabundle-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:46.571589 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.571595    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/488c6d0b-019c-4317-9179-ce5a400bba8f-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:46.571801 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.571604    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/488c6d0b-019c-4317-9179-ce5a400bba8f-kserve-provision-location\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:46.571801 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.571615    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/488c6d0b-019c-4317-9179-ce5a400bba8f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:46.859006 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.858921    2576 generic.go:356] "Generic (PLEG): container finished" podID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerID="d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493" exitCode=0
Sep 16 19:01:46.859006 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.858973    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"
Sep 16 19:01:46.859178 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.859000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" event={"ID":"488c6d0b-019c-4317-9179-ce5a400bba8f","Type":"ContainerDied","Data":"d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493"}
Sep 16 19:01:46.859178 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.859062    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" event={"ID":"488c6d0b-019c-4317-9179-ce5a400bba8f","Type":"ContainerDied","Data":"7f68f673cb561f3381b1b5154478f31c0d607c3526eec2988e7871a04865e051"}
Sep 16 19:01:46.859178 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.859089    2576 scope.go:117] "RemoveContainer" containerID="689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c"
Sep 16 19:01:46.867866 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.867826    2576 scope.go:117] "RemoveContainer" containerID="d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493"
Sep 16 19:01:46.875318 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.875303    2576 scope.go:117] "RemoveContainer" containerID="de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843"
Sep 16 19:01:46.882512 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.882493    2576 scope.go:117] "RemoveContainer" containerID="689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c"
Sep 16 19:01:46.882794 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:46.882773    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c\": container with ID starting with 689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c not found: ID does not exist" containerID="689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c"
Sep 16 19:01:46.882908 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.882804    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c"} err="failed to get container status \"689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c\": rpc error: code = NotFound desc = could not find container \"689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c\": container with ID starting with 689d6974a61789f34219c6bbade98f72a46f1c1eeba7597be633e6d9ca8f740c not found: ID does not exist"
Sep 16 19:01:46.882908 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.882826    2576 scope.go:117] "RemoveContainer" containerID="d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493"
Sep 16 19:01:46.883261 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:46.883244    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493\": container with ID starting with d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493 not found: ID does not exist" containerID="d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493"
Sep 16 19:01:46.883324 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.883267    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493"} err="failed to get container status \"d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493\": rpc error: code = NotFound desc = could not find container \"d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493\": container with ID starting with d7ba874d98cf7c8952804acd1e88457c6392ef27bafc0281965c1d1c5e3e1493 not found: ID does not exist"
Sep 16 19:01:46.883324 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.883283    2576 scope.go:117] "RemoveContainer" containerID="de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843"
Sep 16 19:01:46.883505 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:01:46.883489    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843\": container with ID starting with de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843 not found: ID does not exist" containerID="de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843"
Sep 16 19:01:46.883550 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.883510    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843"} err="failed to get container status \"de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843\": rpc error: code = NotFound desc = could not find container \"de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843\": container with ID starting with de4b1bbd0b4be5ec8ed7178e1e44bc0b11192e281d148fe871c970b28538a843 not found: ID does not exist"
Sep 16 19:01:46.885062 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.885043    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"]
Sep 16 19:01:46.890056 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:46.890037    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv"]
Sep 16 19:01:47.316139 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:47.316099    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-69b99cc798-fhcqv" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:01:48.335382 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:48.335342    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" path="/var/lib/kubelet/pods/488c6d0b-019c-4317-9179-ce5a400bba8f/volumes"
Sep 16 19:01:48.644069 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:48.643992    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:01:49.847777 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:49.847750    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:01:49.848350 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:49.848270    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:01:59.848738 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:01:59.848695    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:02:09.848955 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:02:09.848917    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:02:19.849155 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:02:19.849117    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:02:29.849312 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:02:29.849277    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:04:10.350251 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:04:10.350224    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:04:10.357605 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:04:10.357583    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:09:10.386439 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:09:10.386403    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:09:10.387960 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:09:10.387941    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:10:16.189306 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.189264    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"]
Sep 16 19:10:16.189820 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.189578    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container" containerID="cri-o://09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee" gracePeriod=30
Sep 16 19:10:16.189820 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.189563    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kube-rbac-proxy" containerID="cri-o://a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c" gracePeriod=30
Sep 16 19:10:16.255822 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.255783    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"]
Sep 16 19:10:16.256218 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256203    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="storage-initializer"
Sep 16 19:10:16.256266 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256220    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="storage-initializer"
Sep 16 19:10:16.256266 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256228    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kube-rbac-proxy"
Sep 16 19:10:16.256266 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256234    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kube-rbac-proxy"
Sep 16 19:10:16.256266 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256243    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container"
Sep 16 19:10:16.256266 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256250    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container"
Sep 16 19:10:16.256266 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256261    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kube-rbac-proxy"
Sep 16 19:10:16.256266 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256266    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kube-rbac-proxy"
Sep 16 19:10:16.256479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256280    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container"
Sep 16 19:10:16.256479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256286    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container"
Sep 16 19:10:16.256479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256293    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="storage-initializer"
Sep 16 19:10:16.256479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256299    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="storage-initializer"
Sep 16 19:10:16.256479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256354    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kube-rbac-proxy"
Sep 16 19:10:16.256479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256362    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="348afb3f-512c-4d84-99af-d4480160c6c8" containerName="kserve-container"
Sep 16 19:10:16.256479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256369    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kube-rbac-proxy"
Sep 16 19:10:16.256479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.256376    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="488c6d0b-019c-4317-9179-ce5a400bba8f" containerName="kserve-container"
Sep 16 19:10:16.259613 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.259595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.262016 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.261977    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-016d3-kube-rbac-proxy-sar-config\""
Sep 16 19:10:16.262016 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.261986    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-016d3-predictor-serving-cert\""
Sep 16 19:10:16.269016 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.268989    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"]
Sep 16 19:10:16.336922 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.336887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.337109 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.336978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfmv\" (UniqueName: \"kubernetes.io/projected/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-api-access-ctfmv\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.337109 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.337023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.437934 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.437883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.438114 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.437987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfmv\" (UniqueName: \"kubernetes.io/projected/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-api-access-ctfmv\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.438114 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.438022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.438213 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:10:16.438134    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-016d3-predictor-serving-cert: secret "error-404-isvc-016d3-predictor-serving-cert" not found
Sep 16 19:10:16.438213 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:10:16.438196    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls podName:fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1 nodeName:}" failed. No retries permitted until 2026-09-16 19:10:16.93817604 +0000 UTC m=+1567.132039691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls") pod "error-404-isvc-016d3-predictor-847bb5744-9dgz5" (UID: "fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1") : secret "error-404-isvc-016d3-predictor-serving-cert" not found
Sep 16 19:10:16.438631 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.438612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.447019 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.446938    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfmv\" (UniqueName: \"kubernetes.io/projected/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-api-access-ctfmv\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.942586 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:16.942552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:16.943154 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:10:16.942903    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-016d3-predictor-serving-cert: secret "error-404-isvc-016d3-predictor-serving-cert" not found
Sep 16 19:10:16.943264 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:10:16.943239    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls podName:fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1 nodeName:}" failed. No retries permitted until 2026-09-16 19:10:17.943204074 +0000 UTC m=+1568.137067730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls") pod "error-404-isvc-016d3-predictor-847bb5744-9dgz5" (UID: "fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1") : secret "error-404-isvc-016d3-predictor-serving-cert" not found
Sep 16 19:10:17.188378 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:17.188344    2576 generic.go:356] "Generic (PLEG): container finished" podID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerID="a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c" exitCode=2
Sep 16 19:10:17.188553 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:17.188411    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" event={"ID":"4f0b0141-408d-4222-b40f-6ca6c655f912","Type":"ContainerDied","Data":"a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c"}
Sep 16 19:10:17.951770 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:17.951715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:17.954318 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:17.954291    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls\") pod \"error-404-isvc-016d3-predictor-847bb5744-9dgz5\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") " pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:18.070814 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:18.070765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:18.344031 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:18.343970    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"]
Sep 16 19:10:18.346342 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:10:18.346309    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8a8d73_e46b_4bb3_b6a7_fc9d10f1f4f1.slice/crio-2e9e0bb98f39eed8c2c40197ebadf55612395b1afb3df0f11af99cefe03b79ae WatchSource:0}: Error finding container 2e9e0bb98f39eed8c2c40197ebadf55612395b1afb3df0f11af99cefe03b79ae: Status 404 returned error can't find the container with id 2e9e0bb98f39eed8c2c40197ebadf55612395b1afb3df0f11af99cefe03b79ae
Sep 16 19:10:18.638994 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:18.638898    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:10:18.644304 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:18.644270    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:10:19.199566 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.199530    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" event={"ID":"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1","Type":"ContainerStarted","Data":"4a2956ca71e5f4ae513aba17e73927f486d787321a70216ef09d6d359d0b3687"}
Sep 16 19:10:19.199964 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.199598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" event={"ID":"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1","Type":"ContainerStarted","Data":"f87ed6e1e79605aea81b6b3c4b7da301846f28ce14416bd91ef8df6cb4164d40"}
Sep 16 19:10:19.199964 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.199629    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:19.199964 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.199650    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" event={"ID":"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1","Type":"ContainerStarted","Data":"2e9e0bb98f39eed8c2c40197ebadf55612395b1afb3df0f11af99cefe03b79ae"}
Sep 16 19:10:19.217003 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.216947    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" podStartSLOduration=3.216928205 podStartE2EDuration="3.216928205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:19.216169881 +0000 UTC m=+1569.410033556" watchObservedRunningTime="2026-09-16 19:10:19.216928205 +0000 UTC m=+1569.410791876"
Sep 16 19:10:19.758245 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.758217    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:10:19.869914 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.869816    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-rbac-proxy-sar-config\") pod \"4f0b0141-408d-4222-b40f-6ca6c655f912\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") "
Sep 16 19:10:19.869914 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.869885    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0b0141-408d-4222-b40f-6ca6c655f912-proxy-tls\") pod \"4f0b0141-408d-4222-b40f-6ca6c655f912\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") "
Sep 16 19:10:19.869914 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.869912    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmcwf\" (UniqueName: \"kubernetes.io/projected/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-api-access-zmcwf\") pod \"4f0b0141-408d-4222-b40f-6ca6c655f912\" (UID: \"4f0b0141-408d-4222-b40f-6ca6c655f912\") "
Sep 16 19:10:19.870193 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.870166    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f0b0141-408d-4222-b40f-6ca6c655f912" (UID: "4f0b0141-408d-4222-b40f-6ca6c655f912"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:19.872109 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.872064    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-api-access-zmcwf" (OuterVolumeSpecName: "kube-api-access-zmcwf") pod "4f0b0141-408d-4222-b40f-6ca6c655f912" (UID: "4f0b0141-408d-4222-b40f-6ca6c655f912"). InnerVolumeSpecName "kube-api-access-zmcwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:19.872109 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.872091    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0b0141-408d-4222-b40f-6ca6c655f912-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f0b0141-408d-4222-b40f-6ca6c655f912" (UID: "4f0b0141-408d-4222-b40f-6ca6c655f912"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:19.970569 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.970537    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:19.970569 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.970564    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0b0141-408d-4222-b40f-6ca6c655f912-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:19.970569 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:19.970573    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmcwf\" (UniqueName: \"kubernetes.io/projected/4f0b0141-408d-4222-b40f-6ca6c655f912-kube-api-access-zmcwf\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:20.205756 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.205718    2576 generic.go:356] "Generic (PLEG): container finished" podID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerID="09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee" exitCode=0
Sep 16 19:10:20.206203 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.205774    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"
Sep 16 19:10:20.206203 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.205801    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" event={"ID":"4f0b0141-408d-4222-b40f-6ca6c655f912","Type":"ContainerDied","Data":"09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee"}
Sep 16 19:10:20.206203 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.205897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt" event={"ID":"4f0b0141-408d-4222-b40f-6ca6c655f912","Type":"ContainerDied","Data":"d53f711fa83b2e64c70eeb67840e24e0772340c686f27415240364a504605bef"}
Sep 16 19:10:20.206203 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.205929    2576 scope.go:117] "RemoveContainer" containerID="a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c"
Sep 16 19:10:20.206397 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.206326    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:20.207906 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.207872    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:10:20.215302 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.215280    2576 scope.go:117] "RemoveContainer" containerID="09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee"
Sep 16 19:10:20.223527 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.223499    2576 scope.go:117] "RemoveContainer" containerID="a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c"
Sep 16 19:10:20.223944 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:10:20.223867    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c\": container with ID starting with a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c not found: ID does not exist" containerID="a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c"
Sep 16 19:10:20.224029 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.223971    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c"} err="failed to get container status \"a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c\": rpc error: code = NotFound desc = could not find container \"a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c\": container with ID starting with a127d17381f3fa51c36febf5a2ad1d308a23cad83486a84e9e6d5693dc82a54c not found: ID does not exist"
Sep 16 19:10:20.224029 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.224000    2576 scope.go:117] "RemoveContainer" containerID="09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee"
Sep 16 19:10:20.224397 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:10:20.224367    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee\": container with ID starting with 09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee not found: ID does not exist" containerID="09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee"
Sep 16 19:10:20.224513 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.224407    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee"} err="failed to get container status \"09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee\": rpc error: code = NotFound desc = could not find container \"09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee\": container with ID starting with 09a139514c9e6f3edb778f09c2061b54a52b82e5b4ce5eb252f0f182f08432ee not found: ID does not exist"
Sep 16 19:10:20.226263 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.226238    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"]
Sep 16 19:10:20.228688 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.228662    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93efe-predictor-6486b7df5b-vl2jt"]
Sep 16 19:10:20.336470 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:20.336429    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" path="/var/lib/kubelet/pods/4f0b0141-408d-4222-b40f-6ca6c655f912/volumes"
Sep 16 19:10:21.212158 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:21.212121    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:10:26.216300 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:26.216271    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:10:26.216745 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:26.216719    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:10:36.217479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:36.217436    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:10:46.217392 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:46.217352    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:10:55.900180 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:55.900141    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"]
Sep 16 19:10:55.900614 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:55.900388    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container" containerID="cri-o://4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d" gracePeriod=30
Sep 16 19:10:55.900614 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:55.900471    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kube-rbac-proxy" containerID="cri-o://b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2" gracePeriod=30
Sep 16 19:10:56.105024 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.104988    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"]
Sep 16 19:10:56.105420 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.105407    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container"
Sep 16 19:10:56.105462 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.105422    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container"
Sep 16 19:10:56.105462 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.105431    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kube-rbac-proxy"
Sep 16 19:10:56.105462 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.105437    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kube-rbac-proxy"
Sep 16 19:10:56.105556 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.105500    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kserve-container"
Sep 16 19:10:56.105556 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.105509    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0b0141-408d-4222-b40f-6ca6c655f912" containerName="kube-rbac-proxy"
Sep 16 19:10:56.108696 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.108671    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.110971 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.110941    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7ea69-predictor-serving-cert\""
Sep 16 19:10:56.110971 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.110947    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7ea69-kube-rbac-proxy-sar-config\""
Sep 16 19:10:56.118567 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.118535    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"]
Sep 16 19:10:56.195435 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.195398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfsw7\" (UniqueName: \"kubernetes.io/projected/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-api-access-tfsw7\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.195645 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.195454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.195645 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.195528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d93dc9af-4a76-4658-9a54-0907e8b63184-proxy-tls\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.216723 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.216679    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:10:56.296566 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.296521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d93dc9af-4a76-4658-9a54-0907e8b63184-proxy-tls\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.296773 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.296614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfsw7\" (UniqueName: \"kubernetes.io/projected/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-api-access-tfsw7\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.296773 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.296664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.296773 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:10:56.296701    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-serving-cert: secret "error-404-isvc-7ea69-predictor-serving-cert" not found
Sep 16 19:10:56.296995 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:10:56.296794    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93dc9af-4a76-4658-9a54-0907e8b63184-proxy-tls podName:d93dc9af-4a76-4658-9a54-0907e8b63184 nodeName:}" failed. No retries permitted until 2026-09-16 19:10:56.796769618 +0000 UTC m=+1606.990633274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d93dc9af-4a76-4658-9a54-0907e8b63184-proxy-tls") pod "error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" (UID: "d93dc9af-4a76-4658-9a54-0907e8b63184") : secret "error-404-isvc-7ea69-predictor-serving-cert" not found
Sep 16 19:10:56.297385 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.297360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.304958 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.304928    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfsw7\" (UniqueName: \"kubernetes.io/projected/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-api-access-tfsw7\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.387724 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.387681    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a563277-f8d3-4453-8449-670c1979f306" containerID="b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2" exitCode=2
Sep 16 19:10:56.387935 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.387755    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" event={"ID":"5a563277-f8d3-4453-8449-670c1979f306","Type":"ContainerDied","Data":"b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2"}
Sep 16 19:10:56.801000 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.800944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d93dc9af-4a76-4658-9a54-0907e8b63184-proxy-tls\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:56.803408 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:56.803384    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d93dc9af-4a76-4658-9a54-0907e8b63184-proxy-tls\") pod \"error-404-isvc-7ea69-predictor-758bc9476d-hlmqm\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") " pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:57.019756 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:57.019719    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:57.294732 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:57.294703    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"]
Sep 16 19:10:57.297813 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:10:57.297784    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93dc9af_4a76_4658_9a54_0907e8b63184.slice/crio-5fd5af97886d0f0f8b9e2503d7373c9137d82db397f5255292a860c0888c86a2 WatchSource:0}: Error finding container 5fd5af97886d0f0f8b9e2503d7373c9137d82db397f5255292a860c0888c86a2: Status 404 returned error can't find the container with id 5fd5af97886d0f0f8b9e2503d7373c9137d82db397f5255292a860c0888c86a2
Sep 16 19:10:57.394424 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:57.394387    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" event={"ID":"d93dc9af-4a76-4658-9a54-0907e8b63184","Type":"ContainerStarted","Data":"78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6"}
Sep 16 19:10:57.394597 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:57.394449    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" event={"ID":"d93dc9af-4a76-4658-9a54-0907e8b63184","Type":"ContainerStarted","Data":"5fd5af97886d0f0f8b9e2503d7373c9137d82db397f5255292a860c0888c86a2"}
Sep 16 19:10:58.401560 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:58.401519    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" event={"ID":"d93dc9af-4a76-4658-9a54-0907e8b63184","Type":"ContainerStarted","Data":"a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2"}
Sep 16 19:10:58.401976 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:58.401729    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:58.418477 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:58.418430    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" podStartSLOduration=2.418410849 podStartE2EDuration="2.418410849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:58.417383184 +0000 UTC m=+1608.611246855" watchObservedRunningTime="2026-09-16 19:10:58.418410849 +0000 UTC m=+1608.612274511"
Sep 16 19:10:59.406816 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:59.406786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:10:59.408328 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:59.408291    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:10:59.844138 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:59.844091    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 16 19:10:59.848532 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:10:59.848494    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:11:00.163147 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.163118    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:11:00.230741 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.230685    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a563277-f8d3-4453-8449-670c1979f306-proxy-tls\") pod \"5a563277-f8d3-4453-8449-670c1979f306\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") "
Sep 16 19:11:00.230949 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.230848    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a563277-f8d3-4453-8449-670c1979f306-kube-rbac-proxy-sar-config\") pod \"5a563277-f8d3-4453-8449-670c1979f306\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") "
Sep 16 19:11:00.230949 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.230875    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87js4\" (UniqueName: \"kubernetes.io/projected/5a563277-f8d3-4453-8449-670c1979f306-kube-api-access-87js4\") pod \"5a563277-f8d3-4453-8449-670c1979f306\" (UID: \"5a563277-f8d3-4453-8449-670c1979f306\") "
Sep 16 19:11:00.231247 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.231214    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a563277-f8d3-4453-8449-670c1979f306-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a563277-f8d3-4453-8449-670c1979f306" (UID: "5a563277-f8d3-4453-8449-670c1979f306"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:00.232896 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.232875    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a563277-f8d3-4453-8449-670c1979f306-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a563277-f8d3-4453-8449-670c1979f306" (UID: "5a563277-f8d3-4453-8449-670c1979f306"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:00.232975 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.232899    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a563277-f8d3-4453-8449-670c1979f306-kube-api-access-87js4" (OuterVolumeSpecName: "kube-api-access-87js4") pod "5a563277-f8d3-4453-8449-670c1979f306" (UID: "5a563277-f8d3-4453-8449-670c1979f306"). InnerVolumeSpecName "kube-api-access-87js4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:00.332056 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.332008    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a563277-f8d3-4453-8449-670c1979f306-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:00.332226 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.332060    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87js4\" (UniqueName: \"kubernetes.io/projected/5a563277-f8d3-4453-8449-670c1979f306-kube-api-access-87js4\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:00.332226 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.332084    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a563277-f8d3-4453-8449-670c1979f306-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:00.413635 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.413601    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a563277-f8d3-4453-8449-670c1979f306" containerID="4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d" exitCode=0
Sep 16 19:11:00.414095 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.413655    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"
Sep 16 19:11:00.414095 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.413686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" event={"ID":"5a563277-f8d3-4453-8449-670c1979f306","Type":"ContainerDied","Data":"4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d"}
Sep 16 19:11:00.414095 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.413757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd" event={"ID":"5a563277-f8d3-4453-8449-670c1979f306","Type":"ContainerDied","Data":"25379d40f5da32db781b5ad68ad19742515cdd181c51940cf3ebba839648d47d"}
Sep 16 19:11:00.414095 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.413783    2576 scope.go:117] "RemoveContainer" containerID="b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2"
Sep 16 19:11:00.414371 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.414348    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:11:00.422505 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.422485    2576 scope.go:117] "RemoveContainer" containerID="4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d"
Sep 16 19:11:00.429551 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.429524    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"]
Sep 16 19:11:00.431211 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.431194    2576 scope.go:117] "RemoveContainer" containerID="b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2"
Sep 16 19:11:00.431756 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:11:00.431606    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2\": container with ID starting with b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2 not found: ID does not exist" containerID="b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2"
Sep 16 19:11:00.431756 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.431645    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2"} err="failed to get container status \"b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2\": rpc error: code = NotFound desc = could not find container \"b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2\": container with ID starting with b3b12ba7cf80db21aabf2476ce41445dc9a022e190009ab63c73ae8940e079c2 not found: ID does not exist"
Sep 16 19:11:00.431756 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.431669    2576 scope.go:117] "RemoveContainer" containerID="4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d"
Sep 16 19:11:00.432117 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:11:00.432039    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d\": container with ID starting with 4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d not found: ID does not exist" containerID="4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d"
Sep 16 19:11:00.432117 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.432072    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d"} err="failed to get container status \"4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d\": rpc error: code = NotFound desc = could not find container \"4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d\": container with ID starting with 4fcf412c5a58eb2904a0a5b9056f4b253e0c92a316536c4980ae84562ae1595d not found: ID does not exist"
Sep 16 19:11:00.433881 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:00.433861    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb9a3-predictor-84cdb6dd56-zxpfd"]
Sep 16 19:11:02.335589 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:02.335510    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a563277-f8d3-4453-8449-670c1979f306" path="/var/lib/kubelet/pods/5a563277-f8d3-4453-8449-670c1979f306/volumes"
Sep 16 19:11:05.419548 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:05.419517    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:11:05.419971 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:05.419947    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:11:06.217727 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:06.217691    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:11:15.420685 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:15.420643    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:11:25.420946 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:25.420900    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:11:26.434324 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.434285    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"]
Sep 16 19:11:26.434806 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.434596    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container" containerID="cri-o://f87ed6e1e79605aea81b6b3c4b7da301846f28ce14416bd91ef8df6cb4164d40" gracePeriod=30
Sep 16 19:11:26.434806 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.434627    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kube-rbac-proxy" containerID="cri-o://4a2956ca71e5f4ae513aba17e73927f486d787321a70216ef09d6d359d0b3687" gracePeriod=30
Sep 16 19:11:26.660099 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.660051    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"]
Sep 16 19:11:26.660694 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.660664    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kube-rbac-proxy"
Sep 16 19:11:26.660694 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.660689    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kube-rbac-proxy"
Sep 16 19:11:26.660950 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.660709    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container"
Sep 16 19:11:26.660950 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.660717    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container"
Sep 16 19:11:26.660950 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.660782    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kserve-container"
Sep 16 19:11:26.660950 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.660794    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a563277-f8d3-4453-8449-670c1979f306" containerName="kube-rbac-proxy"
Sep 16 19:11:26.664429 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.664365    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:26.667030 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.667004    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6b79a-kube-rbac-proxy-sar-config\""
Sep 16 19:11:26.667264 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.667246    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6b79a-predictor-serving-cert\""
Sep 16 19:11:26.674094 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.674066    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"]
Sep 16 19:11:26.768578 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.768471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9jp\" (UniqueName: \"kubernetes.io/projected/82ec7de5-faf1-4695-b270-64a356fd6423-kube-api-access-fj9jp\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:26.768777 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.768587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ec7de5-faf1-4695-b270-64a356fd6423-proxy-tls\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:26.768777 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.768697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82ec7de5-faf1-4695-b270-64a356fd6423-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:26.869420 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.869368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82ec7de5-faf1-4695-b270-64a356fd6423-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:26.869634 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.869451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9jp\" (UniqueName: \"kubernetes.io/projected/82ec7de5-faf1-4695-b270-64a356fd6423-kube-api-access-fj9jp\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:26.869634 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.869523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ec7de5-faf1-4695-b270-64a356fd6423-proxy-tls\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:26.869757 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:11:26.869637    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-serving-cert: secret "error-404-isvc-6b79a-predictor-serving-cert" not found
Sep 16 19:11:26.869757 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:11:26.869697    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ec7de5-faf1-4695-b270-64a356fd6423-proxy-tls podName:82ec7de5-faf1-4695-b270-64a356fd6423 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:27.369678637 +0000 UTC m=+1637.563542293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/82ec7de5-faf1-4695-b270-64a356fd6423-proxy-tls") pod "error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" (UID: "82ec7de5-faf1-4695-b270-64a356fd6423") : secret "error-404-isvc-6b79a-predictor-serving-cert" not found
Sep 16 19:11:26.870101 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.870080    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82ec7de5-faf1-4695-b270-64a356fd6423-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:26.879915 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:26.879879    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9jp\" (UniqueName: \"kubernetes.io/projected/82ec7de5-faf1-4695-b270-64a356fd6423-kube-api-access-fj9jp\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:27.374192 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:27.374139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ec7de5-faf1-4695-b270-64a356fd6423-proxy-tls\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:27.376653 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:27.376629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ec7de5-faf1-4695-b270-64a356fd6423-proxy-tls\") pod \"error-404-isvc-6b79a-predictor-558c7754d6-5xsjk\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") " pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:27.549106 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:27.549070    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerID="4a2956ca71e5f4ae513aba17e73927f486d787321a70216ef09d6d359d0b3687" exitCode=2
Sep 16 19:11:27.549587 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:27.549141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" event={"ID":"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1","Type":"ContainerDied","Data":"4a2956ca71e5f4ae513aba17e73927f486d787321a70216ef09d6d359d0b3687"}
Sep 16 19:11:27.579369 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:27.579335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:27.853168 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:27.853130    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"]
Sep 16 19:11:27.856925 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:11:27.856888    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ec7de5_faf1_4695_b270_64a356fd6423.slice/crio-7c81f5099a55982a95ce7ea82ecb40591756e2272de72a97e3f9be03d1d3e624 WatchSource:0}: Error finding container 7c81f5099a55982a95ce7ea82ecb40591756e2272de72a97e3f9be03d1d3e624: Status 404 returned error can't find the container with id 7c81f5099a55982a95ce7ea82ecb40591756e2272de72a97e3f9be03d1d3e624
Sep 16 19:11:28.556801 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:28.556750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" event={"ID":"82ec7de5-faf1-4695-b270-64a356fd6423","Type":"ContainerStarted","Data":"7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041"}
Sep 16 19:11:28.557275 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:28.556832    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" event={"ID":"82ec7de5-faf1-4695-b270-64a356fd6423","Type":"ContainerStarted","Data":"49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a"}
Sep 16 19:11:28.557275 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:28.556889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" event={"ID":"82ec7de5-faf1-4695-b270-64a356fd6423","Type":"ContainerStarted","Data":"7c81f5099a55982a95ce7ea82ecb40591756e2272de72a97e3f9be03d1d3e624"}
Sep 16 19:11:28.557275 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:28.556922    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:28.573252 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:28.573200    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" podStartSLOduration=2.573184355 podStartE2EDuration="2.573184355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:28.572007647 +0000 UTC m=+1638.765871319" watchObservedRunningTime="2026-09-16 19:11:28.573184355 +0000 UTC m=+1638.767048021"
Sep 16 19:11:29.561813 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:29.561778    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:29.563236 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:29.563205    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:30.568394 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.568362    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerID="f87ed6e1e79605aea81b6b3c4b7da301846f28ce14416bd91ef8df6cb4164d40" exitCode=0
Sep 16 19:11:30.568874 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.568413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" event={"ID":"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1","Type":"ContainerDied","Data":"f87ed6e1e79605aea81b6b3c4b7da301846f28ce14416bd91ef8df6cb4164d40"}
Sep 16 19:11:30.568952 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.568896    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:30.591725 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.591693    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:11:30.701360 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.701271    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctfmv\" (UniqueName: \"kubernetes.io/projected/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-api-access-ctfmv\") pod \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") "
Sep 16 19:11:30.701360 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.701317    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-rbac-proxy-sar-config\") pod \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") "
Sep 16 19:11:30.701644 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.701436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls\") pod \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\" (UID: \"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1\") "
Sep 16 19:11:30.701721 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.701689    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" (UID: "fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:30.703774 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.703745    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-api-access-ctfmv" (OuterVolumeSpecName: "kube-api-access-ctfmv") pod "fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" (UID: "fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1"). InnerVolumeSpecName "kube-api-access-ctfmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:30.703923 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.703886    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" (UID: "fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:30.803082 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.803047    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfmv\" (UniqueName: \"kubernetes.io/projected/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-api-access-ctfmv\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:30.803082 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.803075    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:30.803082 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:30.803086    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:31.575174 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:31.575139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5" event={"ID":"fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1","Type":"ContainerDied","Data":"2e9e0bb98f39eed8c2c40197ebadf55612395b1afb3df0f11af99cefe03b79ae"}
Sep 16 19:11:31.575589 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:31.575232    2576 scope.go:117] "RemoveContainer" containerID="4a2956ca71e5f4ae513aba17e73927f486d787321a70216ef09d6d359d0b3687"
Sep 16 19:11:31.575589 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:31.575155    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"
Sep 16 19:11:31.584702 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:31.584654    2576 scope.go:117] "RemoveContainer" containerID="f87ed6e1e79605aea81b6b3c4b7da301846f28ce14416bd91ef8df6cb4164d40"
Sep 16 19:11:31.598641 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:31.598610    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"]
Sep 16 19:11:31.600365 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:31.600338    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-016d3-predictor-847bb5744-9dgz5"]
Sep 16 19:11:32.335449 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:32.335412    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" path="/var/lib/kubelet/pods/fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1/volumes"
Sep 16 19:11:35.420808 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:35.420756    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:11:35.573406 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:35.573372    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:11:35.573947 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:35.573917    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:45.420998 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:45.420966    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:11:45.574458 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:45.574415    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:11:55.574083 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:11:55.574041    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:12:05.574465 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:05.574420    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:12:06.195985 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.195950    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"]
Sep 16 19:12:06.196238 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.196215    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container" containerID="cri-o://78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6" gracePeriod=30
Sep 16 19:12:06.196331 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.196271    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kube-rbac-proxy" containerID="cri-o://a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2" gracePeriod=30
Sep 16 19:12:06.433402 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.433365    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"]
Sep 16 19:12:06.433774 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.433761    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kube-rbac-proxy"
Sep 16 19:12:06.433774 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.433775    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kube-rbac-proxy"
Sep 16 19:12:06.433893 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.433786    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container"
Sep 16 19:12:06.433893 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.433792    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container"
Sep 16 19:12:06.433893 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.433877    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kube-rbac-proxy"
Sep 16 19:12:06.433893 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.433891    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8a8d73-e46b-4bb3-b6a7-fc9d10f1f4f1" containerName="kserve-container"
Sep 16 19:12:06.437098 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.437077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.440053 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.440030    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f025b-predictor-serving-cert\""
Sep 16 19:12:06.440396 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.440382    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f025b-kube-rbac-proxy-sar-config\""
Sep 16 19:12:06.457045 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.456956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"]
Sep 16 19:12:06.509253 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.509205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") " pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.509437 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.509284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppv67\" (UniqueName: \"kubernetes.io/projected/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-api-access-ppv67\") pod \"error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") " pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.509437 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.509311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-proxy-tls\") pod \"error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") " pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.610541 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.610493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv67\" (UniqueName: \"kubernetes.io/projected/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-api-access-ppv67\") pod \"error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") " pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.610541 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.610539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-proxy-tls\") pod \"error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") " pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.611008 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.610597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") " pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.611262 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.611244    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") " pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.613170 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.613149    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-proxy-tls\") pod \"error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") " pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.620144 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.620111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppv67\" (UniqueName: \"kubernetes.io/projected/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-api-access-ppv67\") pod \"error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") " pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:06.743730 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.743637    2576 generic.go:356] "Generic (PLEG): container finished" podID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerID="a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2" exitCode=2
Sep 16 19:12:06.743730 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.743716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" event={"ID":"d93dc9af-4a76-4658-9a54-0907e8b63184","Type":"ContainerDied","Data":"a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2"}
Sep 16 19:12:06.747921 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:06.747886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:07.034164 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:07.034071    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"]
Sep 16 19:12:07.037263 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:12:07.037232    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0017f0e9_610e_4d81_afee_8dfd9d1e2bfd.slice/crio-ee4f9c080e63476c6ec4cbe2af7a5fcc1aae91eef671c1c825ef42013c5f0a93 WatchSource:0}: Error finding container ee4f9c080e63476c6ec4cbe2af7a5fcc1aae91eef671c1c825ef42013c5f0a93: Status 404 returned error can't find the container with id ee4f9c080e63476c6ec4cbe2af7a5fcc1aae91eef671c1c825ef42013c5f0a93
Sep 16 19:12:07.751051 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:07.751013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" event={"ID":"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd","Type":"ContainerStarted","Data":"346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8"}
Sep 16 19:12:07.751484 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:07.751088    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" event={"ID":"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd","Type":"ContainerStarted","Data":"cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30"}
Sep 16 19:12:07.751484 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:07.751120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" event={"ID":"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd","Type":"ContainerStarted","Data":"ee4f9c080e63476c6ec4cbe2af7a5fcc1aae91eef671c1c825ef42013c5f0a93"}
Sep 16 19:12:07.751484 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:07.751213    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:07.770389 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:07.770331    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" podStartSLOduration=1.7703125750000002 podStartE2EDuration="1.770312575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:07.769315368 +0000 UTC m=+1677.963179054" watchObservedRunningTime="2026-09-16 19:12:07.770312575 +0000 UTC m=+1677.964176247"
Sep 16 19:12:08.757735 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:08.757696    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:08.759191 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:08.759164    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:12:09.762734 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:09.762695    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:12:10.060597 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.060562    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:12:10.141771 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.141731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfsw7\" (UniqueName: \"kubernetes.io/projected/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-api-access-tfsw7\") pod \"d93dc9af-4a76-4658-9a54-0907e8b63184\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") "
Sep 16 19:12:10.141996 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.141796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d93dc9af-4a76-4658-9a54-0907e8b63184-proxy-tls\") pod \"d93dc9af-4a76-4658-9a54-0907e8b63184\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") "
Sep 16 19:12:10.141996 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.141894    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-rbac-proxy-sar-config\") pod \"d93dc9af-4a76-4658-9a54-0907e8b63184\" (UID: \"d93dc9af-4a76-4658-9a54-0907e8b63184\") "
Sep 16 19:12:10.142258 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.142225    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d93dc9af-4a76-4658-9a54-0907e8b63184" (UID: "d93dc9af-4a76-4658-9a54-0907e8b63184"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:10.144016 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.143987    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d93dc9af-4a76-4658-9a54-0907e8b63184-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d93dc9af-4a76-4658-9a54-0907e8b63184" (UID: "d93dc9af-4a76-4658-9a54-0907e8b63184"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:10.144118 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.144053    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-api-access-tfsw7" (OuterVolumeSpecName: "kube-api-access-tfsw7") pod "d93dc9af-4a76-4658-9a54-0907e8b63184" (UID: "d93dc9af-4a76-4658-9a54-0907e8b63184"). InnerVolumeSpecName "kube-api-access-tfsw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:10.243049 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.243007    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfsw7\" (UniqueName: \"kubernetes.io/projected/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-api-access-tfsw7\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:10.243049 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.243041    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d93dc9af-4a76-4658-9a54-0907e8b63184-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:10.243049 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.243051    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d93dc9af-4a76-4658-9a54-0907e8b63184-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:10.769098 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.769062    2576 generic.go:356] "Generic (PLEG): container finished" podID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerID="78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6" exitCode=0
Sep 16 19:12:10.769486 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.769126    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"
Sep 16 19:12:10.769486 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.769146    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" event={"ID":"d93dc9af-4a76-4658-9a54-0907e8b63184","Type":"ContainerDied","Data":"78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6"}
Sep 16 19:12:10.769486 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.769233    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm" event={"ID":"d93dc9af-4a76-4658-9a54-0907e8b63184","Type":"ContainerDied","Data":"5fd5af97886d0f0f8b9e2503d7373c9137d82db397f5255292a860c0888c86a2"}
Sep 16 19:12:10.769486 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.769261    2576 scope.go:117] "RemoveContainer" containerID="a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2"
Sep 16 19:12:10.777729 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.777706    2576 scope.go:117] "RemoveContainer" containerID="78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6"
Sep 16 19:12:10.785815 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.785784    2576 scope.go:117] "RemoveContainer" containerID="a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2"
Sep 16 19:12:10.786196 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:12:10.786176    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2\": container with ID starting with a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2 not found: ID does not exist" containerID="a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2"
Sep 16 19:12:10.786264 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.786206    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2"} err="failed to get container status \"a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2\": rpc error: code = NotFound desc = could not find container \"a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2\": container with ID starting with a68617673b80b0fe5d431f05188b9274e1700e8c84a55247bb997d0688936ec2 not found: ID does not exist"
Sep 16 19:12:10.786264 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.786226    2576 scope.go:117] "RemoveContainer" containerID="78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6"
Sep 16 19:12:10.786536 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:12:10.786517    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6\": container with ID starting with 78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6 not found: ID does not exist" containerID="78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6"
Sep 16 19:12:10.786596 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.786545    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6"} err="failed to get container status \"78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6\": rpc error: code = NotFound desc = could not find container \"78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6\": container with ID starting with 78635143ec9865429eecf20ac7896b4fecec2d062c598614c3ee4f5ddea788b6 not found: ID does not exist"
Sep 16 19:12:10.789334 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.789308    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"]
Sep 16 19:12:10.793500 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:10.793476    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ea69-predictor-758bc9476d-hlmqm"]
Sep 16 19:12:12.336011 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:12.335972    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" path="/var/lib/kubelet/pods/d93dc9af-4a76-4658-9a54-0907e8b63184/volumes"
Sep 16 19:12:14.767687 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:14.767654    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:12:14.768285 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:14.768258    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:12:15.574639 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:15.574602    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:12:24.768656 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:24.768605    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:12:34.768919 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:34.768816    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:12:44.768716 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:44.768672    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:12:54.769779 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:12:54.769746    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:14:10.418501 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:14:10.418466    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:14:10.420250 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:14:10.420226    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:19:10.455721 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:19:10.455683    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:19:10.458949 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:19:10.458929    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:20:41.539542 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.539499    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"]
Sep 16 19:20:41.540179 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.539779    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container" containerID="cri-o://49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a" gracePeriod=30
Sep 16 19:20:41.540179 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.539817    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kube-rbac-proxy" containerID="cri-o://7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041" gracePeriod=30
Sep 16 19:20:41.666422 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.666386    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"]
Sep 16 19:20:41.666859 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.666827    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container"
Sep 16 19:20:41.666959 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.666862    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container"
Sep 16 19:20:41.666959 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.666882    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kube-rbac-proxy"
Sep 16 19:20:41.666959 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.666891    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kube-rbac-proxy"
Sep 16 19:20:41.667115 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.666991    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kserve-container"
Sep 16 19:20:41.667115 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.667007    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93dc9af-4a76-4658-9a54-0907e8b63184" containerName="kube-rbac-proxy"
Sep 16 19:20:41.670113 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.670090    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.674435 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.674405    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d799f-predictor-serving-cert\""
Sep 16 19:20:41.674599 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.674580    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d799f-kube-rbac-proxy-sar-config\""
Sep 16 19:20:41.707659 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.699723    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"]
Sep 16 19:20:41.759791 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.759750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f0f8ca-d996-4972-b246-ff7ce06c407e-proxy-tls\") pod \"error-404-isvc-d799f-predictor-9f65898db-r9rgc\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") " pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.760011 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.759875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hh7b\" (UniqueName: \"kubernetes.io/projected/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-api-access-9hh7b\") pod \"error-404-isvc-d799f-predictor-9f65898db-r9rgc\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") " pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.760011 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.759926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d799f-predictor-9f65898db-r9rgc\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") " pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.861286 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.861183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d799f-predictor-9f65898db-r9rgc\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") " pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.861286 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.861247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f0f8ca-d996-4972-b246-ff7ce06c407e-proxy-tls\") pod \"error-404-isvc-d799f-predictor-9f65898db-r9rgc\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") " pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.861286 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.861286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hh7b\" (UniqueName: \"kubernetes.io/projected/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-api-access-9hh7b\") pod \"error-404-isvc-d799f-predictor-9f65898db-r9rgc\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") " pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.861956 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.861931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d799f-predictor-9f65898db-r9rgc\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") " pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.863861 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.863817    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f0f8ca-d996-4972-b246-ff7ce06c407e-proxy-tls\") pod \"error-404-isvc-d799f-predictor-9f65898db-r9rgc\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") " pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.874236 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.874213    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hh7b\" (UniqueName: \"kubernetes.io/projected/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-api-access-9hh7b\") pod \"error-404-isvc-d799f-predictor-9f65898db-r9rgc\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") " pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:41.980928 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:41.980892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:42.226020 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:42.225985    2576 generic.go:356] "Generic (PLEG): container finished" podID="82ec7de5-faf1-4695-b270-64a356fd6423" containerID="7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041" exitCode=2
Sep 16 19:20:42.226197 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:42.226056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" event={"ID":"82ec7de5-faf1-4695-b270-64a356fd6423","Type":"ContainerDied","Data":"7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041"}
Sep 16 19:20:42.254552 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:42.254527    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"]
Sep 16 19:20:42.257099 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:20:42.257070    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f0f8ca_d996_4972_b246_ff7ce06c407e.slice/crio-2a21f2ea7a376dea86d1a4be6593455fa7f6871ac60244aca5a0db5c7c04ae77 WatchSource:0}: Error finding container 2a21f2ea7a376dea86d1a4be6593455fa7f6871ac60244aca5a0db5c7c04ae77: Status 404 returned error can't find the container with id 2a21f2ea7a376dea86d1a4be6593455fa7f6871ac60244aca5a0db5c7c04ae77
Sep 16 19:20:43.233102 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:43.233058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" event={"ID":"c7f0f8ca-d996-4972-b246-ff7ce06c407e","Type":"ContainerStarted","Data":"e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c"}
Sep 16 19:20:43.233500 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:43.233119    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" event={"ID":"c7f0f8ca-d996-4972-b246-ff7ce06c407e","Type":"ContainerStarted","Data":"39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d"}
Sep 16 19:20:43.233500 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:43.233138    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" event={"ID":"c7f0f8ca-d996-4972-b246-ff7ce06c407e","Type":"ContainerStarted","Data":"2a21f2ea7a376dea86d1a4be6593455fa7f6871ac60244aca5a0db5c7c04ae77"}
Sep 16 19:20:43.233500 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:43.233181    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:43.255425 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:43.255360    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podStartSLOduration=2.255337982 podStartE2EDuration="2.255337982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:43.253983262 +0000 UTC m=+2193.447846933" watchObservedRunningTime="2026-09-16 19:20:43.255337982 +0000 UTC m=+2193.449201655"
Sep 16 19:20:44.238405 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:44.238372    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:44.239879 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:44.239832    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:20:45.087099 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.087072    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:20:45.190064 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.190030    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj9jp\" (UniqueName: \"kubernetes.io/projected/82ec7de5-faf1-4695-b270-64a356fd6423-kube-api-access-fj9jp\") pod \"82ec7de5-faf1-4695-b270-64a356fd6423\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") "
Sep 16 19:20:45.190244 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.190086    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ec7de5-faf1-4695-b270-64a356fd6423-proxy-tls\") pod \"82ec7de5-faf1-4695-b270-64a356fd6423\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") "
Sep 16 19:20:45.190244 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.190106    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82ec7de5-faf1-4695-b270-64a356fd6423-kube-rbac-proxy-sar-config\") pod \"82ec7de5-faf1-4695-b270-64a356fd6423\" (UID: \"82ec7de5-faf1-4695-b270-64a356fd6423\") "
Sep 16 19:20:45.190538 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.190513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ec7de5-faf1-4695-b270-64a356fd6423-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82ec7de5-faf1-4695-b270-64a356fd6423" (UID: "82ec7de5-faf1-4695-b270-64a356fd6423"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:45.192248 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.192222    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ec7de5-faf1-4695-b270-64a356fd6423-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82ec7de5-faf1-4695-b270-64a356fd6423" (UID: "82ec7de5-faf1-4695-b270-64a356fd6423"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:45.192303 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.192245    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ec7de5-faf1-4695-b270-64a356fd6423-kube-api-access-fj9jp" (OuterVolumeSpecName: "kube-api-access-fj9jp") pod "82ec7de5-faf1-4695-b270-64a356fd6423" (UID: "82ec7de5-faf1-4695-b270-64a356fd6423"). InnerVolumeSpecName "kube-api-access-fj9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:45.244606 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.244573    2576 generic.go:356] "Generic (PLEG): container finished" podID="82ec7de5-faf1-4695-b270-64a356fd6423" containerID="49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a" exitCode=0
Sep 16 19:20:45.245045 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.244624    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"
Sep 16 19:20:45.245045 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.244661    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" event={"ID":"82ec7de5-faf1-4695-b270-64a356fd6423","Type":"ContainerDied","Data":"49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a"}
Sep 16 19:20:45.245045 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.244720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk" event={"ID":"82ec7de5-faf1-4695-b270-64a356fd6423","Type":"ContainerDied","Data":"7c81f5099a55982a95ce7ea82ecb40591756e2272de72a97e3f9be03d1d3e624"}
Sep 16 19:20:45.245045 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.244745    2576 scope.go:117] "RemoveContainer" containerID="7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041"
Sep 16 19:20:45.245252 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.245162    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:20:45.254298 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.254278    2576 scope.go:117] "RemoveContainer" containerID="49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a"
Sep 16 19:20:45.262653 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.262632    2576 scope.go:117] "RemoveContainer" containerID="7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041"
Sep 16 19:20:45.262982 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:20:45.262962    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041\": container with ID starting with 7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041 not found: ID does not exist" containerID="7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041"
Sep 16 19:20:45.263079 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.262994    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041"} err="failed to get container status \"7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041\": rpc error: code = NotFound desc = could not find container \"7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041\": container with ID starting with 7d5bb551d16606c1dc6d0fe98006dcd8d7da1e882c6ae4b5377c24e9118ba041 not found: ID does not exist"
Sep 16 19:20:45.263079 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.263017    2576 scope.go:117] "RemoveContainer" containerID="49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a"
Sep 16 19:20:45.263282 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:20:45.263265    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a\": container with ID starting with 49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a not found: ID does not exist" containerID="49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a"
Sep 16 19:20:45.263330 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.263290    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a"} err="failed to get container status \"49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a\": rpc error: code = NotFound desc = could not find container \"49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a\": container with ID starting with 49dc5a1cfc50af9ce5d17d00f62bfcbfc7014de32c7b36e421a209ee2ca8b40a not found: ID does not exist"
Sep 16 19:20:45.266110 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.266085    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"]
Sep 16 19:20:45.270610 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.270585    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6b79a-predictor-558c7754d6-5xsjk"]
Sep 16 19:20:45.291140 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.291107    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj9jp\" (UniqueName: \"kubernetes.io/projected/82ec7de5-faf1-4695-b270-64a356fd6423-kube-api-access-fj9jp\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:20:45.291140 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.291139    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ec7de5-faf1-4695-b270-64a356fd6423-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:20:45.291287 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:45.291150    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82ec7de5-faf1-4695-b270-64a356fd6423-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:20:46.335390 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:46.335352    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" path="/var/lib/kubelet/pods/82ec7de5-faf1-4695-b270-64a356fd6423/volumes"
Sep 16 19:20:50.256144 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:50.256115    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:20:50.256731 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:20:50.256700    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:21:00.257790 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:00.257744    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:21:10.257438 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:10.257384    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:21:20.256792 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:20.256750    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:21:21.083211 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.083175    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"]
Sep 16 19:21:21.083477 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.083450    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container" containerID="cri-o://cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30" gracePeriod=30
Sep 16 19:21:21.083632 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.083492    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kube-rbac-proxy" containerID="cri-o://346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8" gracePeriod=30
Sep 16 19:21:21.437945 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.437908    2576 generic.go:356] "Generic (PLEG): container finished" podID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerID="346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8" exitCode=2
Sep 16 19:21:21.438421 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.437984    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" event={"ID":"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd","Type":"ContainerDied","Data":"346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8"}
Sep 16 19:21:21.467514 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.467475    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"]
Sep 16 19:21:21.467953 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.467935    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container"
Sep 16 19:21:21.468063 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.467956    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container"
Sep 16 19:21:21.468063 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.467971    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kube-rbac-proxy"
Sep 16 19:21:21.468063 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.467979    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kube-rbac-proxy"
Sep 16 19:21:21.468233 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.468111    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kserve-container"
Sep 16 19:21:21.468233 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.468127    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ec7de5-faf1-4695-b270-64a356fd6423" containerName="kube-rbac-proxy"
Sep 16 19:21:21.471453 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.471432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.474093 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.474062    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0263c-kube-rbac-proxy-sar-config\""
Sep 16 19:21:21.474231 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.474160    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0263c-predictor-serving-cert\""
Sep 16 19:21:21.482303 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.482278    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"]
Sep 16 19:21:21.568741 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.568707    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") " pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.568741 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.568745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fghh\" (UniqueName: \"kubernetes.io/projected/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-api-access-7fghh\") pod \"error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") " pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.569004 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.568784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-proxy-tls\") pod \"error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") " pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.669536 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.669500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-proxy-tls\") pod \"error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") " pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.669721 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.669641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") " pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.669721 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.669667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fghh\" (UniqueName: \"kubernetes.io/projected/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-api-access-7fghh\") pod \"error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") " pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.670321 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.670303    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") " pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.672217 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.672194    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-proxy-tls\") pod \"error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") " pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.678877 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.678850    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fghh\" (UniqueName: \"kubernetes.io/projected/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-api-access-7fghh\") pod \"error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") " pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:21.783658 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:21.783558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:22.079297 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:22.079271    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"]
Sep 16 19:21:22.081976 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:21:22.081951    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc735b32_c4ab_4ec4_aaf4_b51852684b5e.slice/crio-24dd9bfdf9e6da8184780b0047c370e9ddc3e9451da9e4fc525a935b65762ffc WatchSource:0}: Error finding container 24dd9bfdf9e6da8184780b0047c370e9ddc3e9451da9e4fc525a935b65762ffc: Status 404 returned error can't find the container with id 24dd9bfdf9e6da8184780b0047c370e9ddc3e9451da9e4fc525a935b65762ffc
Sep 16 19:21:22.445127 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:22.445090    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" event={"ID":"bc735b32-c4ab-4ec4-aaf4-b51852684b5e","Type":"ContainerStarted","Data":"99e46099dca9013dc85798daacf37cb30687e60dc751f597c9d69ab20ec7eb0a"}
Sep 16 19:21:22.445576 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:22.445147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" event={"ID":"bc735b32-c4ab-4ec4-aaf4-b51852684b5e","Type":"ContainerStarted","Data":"b8059629b14ae79dad1f1eb9477036662b0909449b67deec123a352d8f73a996"}
Sep 16 19:21:22.445576 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:22.445172    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" event={"ID":"bc735b32-c4ab-4ec4-aaf4-b51852684b5e","Type":"ContainerStarted","Data":"24dd9bfdf9e6da8184780b0047c370e9ddc3e9451da9e4fc525a935b65762ffc"}
Sep 16 19:21:22.445576 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:22.445199    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:22.463211 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:22.463156    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podStartSLOduration=1.463138396 podStartE2EDuration="1.463138396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:22.462189616 +0000 UTC m=+2232.656053291" watchObservedRunningTime="2026-09-16 19:21:22.463138396 +0000 UTC m=+2232.657002066"
Sep 16 19:21:23.450900 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:23.450870    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:23.452415 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:23.452379    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:21:24.455770 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.455729    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:21:24.736147 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.736120    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:21:24.795277 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.795241    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppv67\" (UniqueName: \"kubernetes.io/projected/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-api-access-ppv67\") pod \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") "
Sep 16 19:21:24.795467 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.795291    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-rbac-proxy-sar-config\") pod \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") "
Sep 16 19:21:24.795534 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.795478    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-proxy-tls\") pod \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\" (UID: \"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd\") "
Sep 16 19:21:24.795606 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.795588    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" (UID: "0017f0e9-610e-4d81-afee-8dfd9d1e2bfd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:24.795734 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.795722    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:21:24.797405 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.797382    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-api-access-ppv67" (OuterVolumeSpecName: "kube-api-access-ppv67") pod "0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" (UID: "0017f0e9-610e-4d81-afee-8dfd9d1e2bfd"). InnerVolumeSpecName "kube-api-access-ppv67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:24.797482 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.797380    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" (UID: "0017f0e9-610e-4d81-afee-8dfd9d1e2bfd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:24.896988 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.896947    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:21:24.896988 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:24.896982    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppv67\" (UniqueName: \"kubernetes.io/projected/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd-kube-api-access-ppv67\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.461970 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.461933    2576 generic.go:356] "Generic (PLEG): container finished" podID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerID="cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30" exitCode=0
Sep 16 19:21:25.462374 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.461986    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"
Sep 16 19:21:25.462374 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.462010    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" event={"ID":"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd","Type":"ContainerDied","Data":"cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30"}
Sep 16 19:21:25.462374 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.462062    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk" event={"ID":"0017f0e9-610e-4d81-afee-8dfd9d1e2bfd","Type":"ContainerDied","Data":"ee4f9c080e63476c6ec4cbe2af7a5fcc1aae91eef671c1c825ef42013c5f0a93"}
Sep 16 19:21:25.462374 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.462091    2576 scope.go:117] "RemoveContainer" containerID="346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8"
Sep 16 19:21:25.473376 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.473327    2576 scope.go:117] "RemoveContainer" containerID="cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30"
Sep 16 19:21:25.481990 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.481968    2576 scope.go:117] "RemoveContainer" containerID="346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8"
Sep 16 19:21:25.482360 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:21:25.482336    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8\": container with ID starting with 346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8 not found: ID does not exist" containerID="346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8"
Sep 16 19:21:25.482464 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.482382    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8"} err="failed to get container status \"346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8\": rpc error: code = NotFound desc = could not find container \"346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8\": container with ID starting with 346173f53c4d3025e73462562841d584d3c20ee6283c91d028c8b3f224d700e8 not found: ID does not exist"
Sep 16 19:21:25.482464 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.482408    2576 scope.go:117] "RemoveContainer" containerID="cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30"
Sep 16 19:21:25.482966 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:21:25.482938    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30\": container with ID starting with cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30 not found: ID does not exist" containerID="cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30"
Sep 16 19:21:25.483044 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.482976    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30"} err="failed to get container status \"cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30\": rpc error: code = NotFound desc = could not find container \"cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30\": container with ID starting with cc220570158949e138ea556ef63214c8a8c34b0ee9b840f281f58b55bf2a7b30 not found: ID does not exist"
Sep 16 19:21:25.484293 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.484273    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"]
Sep 16 19:21:25.491586 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:25.491561    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025b-predictor-76fd9c4cc8-9hfsk"]
Sep 16 19:21:26.335550 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:26.335512    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" path="/var/lib/kubelet/pods/0017f0e9-610e-4d81-afee-8dfd9d1e2bfd/volumes"
Sep 16 19:21:29.461108 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:29.461075    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:21:29.461530 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:29.461506    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:21:30.258190 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:30.258159    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:21:39.462141 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:39.462046    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:21:49.462320 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:49.462273    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:21:51.901347 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:51.901310    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"]
Sep 16 19:21:51.901833 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:51.901561    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container" containerID="cri-o://39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d" gracePeriod=30
Sep 16 19:21:51.901833 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:51.901599    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kube-rbac-proxy" containerID="cri-o://e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c" gracePeriod=30
Sep 16 19:21:52.131934 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.131892    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"]
Sep 16 19:21:52.132387 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.132368    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container"
Sep 16 19:21:52.132387 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.132388    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container"
Sep 16 19:21:52.132547 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.132401    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kube-rbac-proxy"
Sep 16 19:21:52.132547 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.132410    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kube-rbac-proxy"
Sep 16 19:21:52.132547 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.132495    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kube-rbac-proxy"
Sep 16 19:21:52.132547 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.132520    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0017f0e9-610e-4d81-afee-8dfd9d1e2bfd" containerName="kserve-container"
Sep 16 19:21:52.136152 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.136125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.138447 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.138423    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3e3d-predictor-serving-cert\""
Sep 16 19:21:52.138447 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.138446    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3e3d-kube-rbac-proxy-sar-config\""
Sep 16 19:21:52.149140 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.149112    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"]
Sep 16 19:21:52.235145 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.235101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.235342 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.235155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4589076b-8d20-428d-91fb-fc5cb61dc2ee-proxy-tls\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.235342 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.235186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrsc\" (UniqueName: \"kubernetes.io/projected/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-api-access-mmrsc\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.336320 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.336289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.336320 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.336322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4589076b-8d20-428d-91fb-fc5cb61dc2ee-proxy-tls\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.336612 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.336341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrsc\" (UniqueName: \"kubernetes.io/projected/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-api-access-mmrsc\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.336612 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:21:52.336457    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-serving-cert: secret "error-404-isvc-c3e3d-predictor-serving-cert" not found
Sep 16 19:21:52.336612 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:21:52.336541    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4589076b-8d20-428d-91fb-fc5cb61dc2ee-proxy-tls podName:4589076b-8d20-428d-91fb-fc5cb61dc2ee nodeName:}" failed. No retries permitted until 2026-09-16 19:21:52.836521422 +0000 UTC m=+2263.030385076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4589076b-8d20-428d-91fb-fc5cb61dc2ee-proxy-tls") pod "error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" (UID: "4589076b-8d20-428d-91fb-fc5cb61dc2ee") : secret "error-404-isvc-c3e3d-predictor-serving-cert" not found
Sep 16 19:21:52.337108 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.337083    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.345514 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.345483    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrsc\" (UniqueName: \"kubernetes.io/projected/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-api-access-mmrsc\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.594688 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.594593    2576 generic.go:356] "Generic (PLEG): container finished" podID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerID="e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c" exitCode=2
Sep 16 19:21:52.594688 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.594670    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" event={"ID":"c7f0f8ca-d996-4972-b246-ff7ce06c407e","Type":"ContainerDied","Data":"e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c"}
Sep 16 19:21:52.840331 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.840291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4589076b-8d20-428d-91fb-fc5cb61dc2ee-proxy-tls\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:52.842913 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:52.842886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4589076b-8d20-428d-91fb-fc5cb61dc2ee-proxy-tls\") pod \"error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:53.049469 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:53.049425    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:53.324451 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:53.324362    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"]
Sep 16 19:21:53.328234 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:21:53.328208    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4589076b_8d20_428d_91fb_fc5cb61dc2ee.slice/crio-910c825aecd6a3722b12a18b17fcde4e450a38f3e981b272874f5db6ae7a9f08 WatchSource:0}: Error finding container 910c825aecd6a3722b12a18b17fcde4e450a38f3e981b272874f5db6ae7a9f08: Status 404 returned error can't find the container with id 910c825aecd6a3722b12a18b17fcde4e450a38f3e981b272874f5db6ae7a9f08
Sep 16 19:21:53.601512 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:53.601414    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" event={"ID":"4589076b-8d20-428d-91fb-fc5cb61dc2ee","Type":"ContainerStarted","Data":"5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52"}
Sep 16 19:21:53.601512 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:53.601488    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" event={"ID":"4589076b-8d20-428d-91fb-fc5cb61dc2ee","Type":"ContainerStarted","Data":"d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925"}
Sep 16 19:21:53.601701 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:53.601518    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" event={"ID":"4589076b-8d20-428d-91fb-fc5cb61dc2ee","Type":"ContainerStarted","Data":"910c825aecd6a3722b12a18b17fcde4e450a38f3e981b272874f5db6ae7a9f08"}
Sep 16 19:21:53.601701 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:53.601639    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:53.620915 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:53.620859    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" podStartSLOduration=1.620826283 podStartE2EDuration="1.620826283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:53.619171669 +0000 UTC m=+2263.813035344" watchObservedRunningTime="2026-09-16 19:21:53.620826283 +0000 UTC m=+2263.814689955"
Sep 16 19:21:54.606182 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:54.606149    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:21:54.607532 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:54.607500    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:21:55.245954 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.245908    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 16 19:21:55.559260 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.559225    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:21:55.612576 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.612542    2576 generic.go:356] "Generic (PLEG): container finished" podID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerID="39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d" exitCode=0
Sep 16 19:21:55.613070 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.612589    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"
Sep 16 19:21:55.613070 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.612620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" event={"ID":"c7f0f8ca-d996-4972-b246-ff7ce06c407e","Type":"ContainerDied","Data":"39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d"}
Sep 16 19:21:55.613070 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.612695    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc" event={"ID":"c7f0f8ca-d996-4972-b246-ff7ce06c407e","Type":"ContainerDied","Data":"2a21f2ea7a376dea86d1a4be6593455fa7f6871ac60244aca5a0db5c7c04ae77"}
Sep 16 19:21:55.613070 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.612738    2576 scope.go:117] "RemoveContainer" containerID="e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c"
Sep 16 19:21:55.613070 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.613050    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:21:55.621295 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.621274    2576 scope.go:117] "RemoveContainer" containerID="39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d"
Sep 16 19:21:55.629524 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.629503    2576 scope.go:117] "RemoveContainer" containerID="e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c"
Sep 16 19:21:55.629908 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:21:55.629879    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c\": container with ID starting with e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c not found: ID does not exist" containerID="e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c"
Sep 16 19:21:55.629959 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.629925    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c"} err="failed to get container status \"e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c\": rpc error: code = NotFound desc = could not find container \"e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c\": container with ID starting with e13903abb81a7a57d8c29afa86a0dd6af7c4586c8ebba72f2dbd583768de2d4c not found: ID does not exist"
Sep 16 19:21:55.629959 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.629951    2576 scope.go:117] "RemoveContainer" containerID="39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d"
Sep 16 19:21:55.630239 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:21:55.630217    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d\": container with ID starting with 39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d not found: ID does not exist" containerID="39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d"
Sep 16 19:21:55.630299 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.630249    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d"} err="failed to get container status \"39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d\": rpc error: code = NotFound desc = could not find container \"39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d\": container with ID starting with 39f256b95f64675f0a1a2d22842daf99162ad7c68dfc855d6144a2750648db1d not found: ID does not exist"
Sep 16 19:21:55.667411 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.667372    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hh7b\" (UniqueName: \"kubernetes.io/projected/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-api-access-9hh7b\") pod \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") "
Sep 16 19:21:55.667607 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.667429    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-rbac-proxy-sar-config\") pod \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") "
Sep 16 19:21:55.667607 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.667524    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f0f8ca-d996-4972-b246-ff7ce06c407e-proxy-tls\") pod \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\" (UID: \"c7f0f8ca-d996-4972-b246-ff7ce06c407e\") "
Sep 16 19:21:55.667852 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.667816    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7f0f8ca-d996-4972-b246-ff7ce06c407e" (UID: "c7f0f8ca-d996-4972-b246-ff7ce06c407e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:55.669695 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.669666    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-api-access-9hh7b" (OuterVolumeSpecName: "kube-api-access-9hh7b") pod "c7f0f8ca-d996-4972-b246-ff7ce06c407e" (UID: "c7f0f8ca-d996-4972-b246-ff7ce06c407e"). InnerVolumeSpecName "kube-api-access-9hh7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:55.669695 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.669666    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f0f8ca-d996-4972-b246-ff7ce06c407e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7f0f8ca-d996-4972-b246-ff7ce06c407e" (UID: "c7f0f8ca-d996-4972-b246-ff7ce06c407e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:55.768375 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.768333    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f0f8ca-d996-4972-b246-ff7ce06c407e-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:21:55.768375 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.768366    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hh7b\" (UniqueName: \"kubernetes.io/projected/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-api-access-9hh7b\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:21:55.768667 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.768452    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f0f8ca-d996-4972-b246-ff7ce06c407e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:21:55.937978 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.937933    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"]
Sep 16 19:21:55.943235 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:55.943203    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d799f-predictor-9f65898db-r9rgc"]
Sep 16 19:21:56.335390 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:56.335308    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" path="/var/lib/kubelet/pods/c7f0f8ca-d996-4972-b246-ff7ce06c407e/volumes"
Sep 16 19:21:59.462550 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:21:59.462508    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:22:00.617536 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:22:00.617501    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:22:00.618163 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:22:00.618142    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:22:09.462860 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:22:09.462804    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:22:10.618898 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:22:10.618831    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:22:20.618530 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:22:20.618486    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:22:30.618851 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:22:30.618794    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:22:40.619634 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:22:40.619603    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:24:10.487709 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:24:10.487604    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:24:10.492024 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:24:10.491998    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:29:10.518268 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:29:10.518152    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:29:10.523926 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:29:10.523904    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:31:06.804570 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:06.804532    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"]
Sep 16 19:31:06.805108 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:06.804773    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container" containerID="cri-o://d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925" gracePeriod=30
Sep 16 19:31:06.805108 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:06.804820    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kube-rbac-proxy" containerID="cri-o://5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52" gracePeriod=30
Sep 16 19:31:07.293892 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:07.293858    2576 generic.go:356] "Generic (PLEG): container finished" podID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerID="5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52" exitCode=2
Sep 16 19:31:07.294064 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:07.293867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" event={"ID":"4589076b-8d20-428d-91fb-fc5cb61dc2ee","Type":"ContainerDied","Data":"5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52"}
Sep 16 19:31:10.265083 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.265047    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:31:10.309970 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.309883    2576 generic.go:356] "Generic (PLEG): container finished" podID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerID="d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925" exitCode=0
Sep 16 19:31:10.309970 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.309938    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"
Sep 16 19:31:10.309970 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.309937    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" event={"ID":"4589076b-8d20-428d-91fb-fc5cb61dc2ee","Type":"ContainerDied","Data":"d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925"}
Sep 16 19:31:10.310172 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.310004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6" event={"ID":"4589076b-8d20-428d-91fb-fc5cb61dc2ee","Type":"ContainerDied","Data":"910c825aecd6a3722b12a18b17fcde4e450a38f3e981b272874f5db6ae7a9f08"}
Sep 16 19:31:10.310172 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.310043    2576 scope.go:117] "RemoveContainer" containerID="5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52"
Sep 16 19:31:10.318648 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.318622    2576 scope.go:117] "RemoveContainer" containerID="d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925"
Sep 16 19:31:10.326880 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.326860    2576 scope.go:117] "RemoveContainer" containerID="5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52"
Sep 16 19:31:10.327186 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:31:10.327164    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52\": container with ID starting with 5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52 not found: ID does not exist" containerID="5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52"
Sep 16 19:31:10.327269 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.327200    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52"} err="failed to get container status \"5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52\": rpc error: code = NotFound desc = could not find container \"5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52\": container with ID starting with 5c34522b4447f26d7b097b7ba6a71f2f9fa98146a104c587879cc0c37a00ae52 not found: ID does not exist"
Sep 16 19:31:10.327269 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.327225    2576 scope.go:117] "RemoveContainer" containerID="d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925"
Sep 16 19:31:10.327484 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:31:10.327459    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925\": container with ID starting with d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925 not found: ID does not exist" containerID="d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925"
Sep 16 19:31:10.327532 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.327490    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925"} err="failed to get container status \"d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925\": rpc error: code = NotFound desc = could not find container \"d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925\": container with ID starting with d17e3a439d09e4c12ee32dc3840629188c96331ab8f960d797269aa2a719b925 not found: ID does not exist"
Sep 16 19:31:10.347781 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.347746    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmrsc\" (UniqueName: \"kubernetes.io/projected/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-api-access-mmrsc\") pod \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") "
Sep 16 19:31:10.347781 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.347785    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4589076b-8d20-428d-91fb-fc5cb61dc2ee-proxy-tls\") pod \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") "
Sep 16 19:31:10.348051 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.347814    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-rbac-proxy-sar-config\") pod \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\" (UID: \"4589076b-8d20-428d-91fb-fc5cb61dc2ee\") "
Sep 16 19:31:10.348247 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.348222    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4589076b-8d20-428d-91fb-fc5cb61dc2ee" (UID: "4589076b-8d20-428d-91fb-fc5cb61dc2ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:10.349967 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.349941    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-api-access-mmrsc" (OuterVolumeSpecName: "kube-api-access-mmrsc") pod "4589076b-8d20-428d-91fb-fc5cb61dc2ee" (UID: "4589076b-8d20-428d-91fb-fc5cb61dc2ee"). InnerVolumeSpecName "kube-api-access-mmrsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:10.350050 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.350026    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4589076b-8d20-428d-91fb-fc5cb61dc2ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4589076b-8d20-428d-91fb-fc5cb61dc2ee" (UID: "4589076b-8d20-428d-91fb-fc5cb61dc2ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:10.449182 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.449140    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmrsc\" (UniqueName: \"kubernetes.io/projected/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-api-access-mmrsc\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:31:10.449182 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.449173    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4589076b-8d20-428d-91fb-fc5cb61dc2ee-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:31:10.449182 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.449182    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4589076b-8d20-428d-91fb-fc5cb61dc2ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:31:10.630681 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.630644    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"]
Sep 16 19:31:10.636102 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:10.636069    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e3d-predictor-744cfbc744-nk6c6"]
Sep 16 19:31:12.334690 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:31:12.334648    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" path="/var/lib/kubelet/pods/4589076b-8d20-428d-91fb-fc5cb61dc2ee/volumes"
Sep 16 19:34:10.548725 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:34:10.548607    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:34:10.556061 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:34:10.556038    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:38:40.644562 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:40.644527    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"]
Sep 16 19:38:40.645150 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:40.644772    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container" containerID="cri-o://b8059629b14ae79dad1f1eb9477036662b0909449b67deec123a352d8f73a996" gracePeriod=30
Sep 16 19:38:40.645150 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:40.644821    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kube-rbac-proxy" containerID="cri-o://99e46099dca9013dc85798daacf37cb30687e60dc751f597c9d69ab20ec7eb0a" gracePeriod=30
Sep 16 19:38:41.391209 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391167    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bnx75/must-gather-z4j7h"]
Sep 16 19:38:41.391614 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391600    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kube-rbac-proxy"
Sep 16 19:38:41.391657 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391616    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kube-rbac-proxy"
Sep 16 19:38:41.391657 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391640    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container"
Sep 16 19:38:41.391657 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391650    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container"
Sep 16 19:38:41.391754 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391658    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kube-rbac-proxy"
Sep 16 19:38:41.391754 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391664    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kube-rbac-proxy"
Sep 16 19:38:41.391754 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391674    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container"
Sep 16 19:38:41.391754 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391679    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container"
Sep 16 19:38:41.391754 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391741    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kube-rbac-proxy"
Sep 16 19:38:41.391754 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391748    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kube-rbac-proxy"
Sep 16 19:38:41.391970 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391757    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f0f8ca-d996-4972-b246-ff7ce06c407e" containerName="kserve-container"
Sep 16 19:38:41.391970 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.391763    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4589076b-8d20-428d-91fb-fc5cb61dc2ee" containerName="kserve-container"
Sep 16 19:38:41.395040 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.395016    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:38:41.397209 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.397181    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnx75\"/\"openshift-service-ca.crt\""
Sep 16 19:38:41.397342 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.397181    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bnx75\"/\"default-dockercfg-tn5cq\""
Sep 16 19:38:41.397811 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.397796    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnx75\"/\"kube-root-ca.crt\""
Sep 16 19:38:41.403845 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.403805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnx75/must-gather-z4j7h"]
Sep 16 19:38:41.405416 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.405390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hxx\" (UniqueName: \"kubernetes.io/projected/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-kube-api-access-x7hxx\") pod \"must-gather-z4j7h\" (UID: \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\") " pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:38:41.405537 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.405448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-must-gather-output\") pod \"must-gather-z4j7h\" (UID: \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\") " pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:38:41.442752 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.442716    2576 generic.go:356] "Generic (PLEG): container finished" podID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerID="99e46099dca9013dc85798daacf37cb30687e60dc751f597c9d69ab20ec7eb0a" exitCode=2
Sep 16 19:38:41.442936 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.442781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" event={"ID":"bc735b32-c4ab-4ec4-aaf4-b51852684b5e","Type":"ContainerDied","Data":"99e46099dca9013dc85798daacf37cb30687e60dc751f597c9d69ab20ec7eb0a"}
Sep 16 19:38:41.506026 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.505989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-must-gather-output\") pod \"must-gather-z4j7h\" (UID: \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\") " pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:38:41.506181 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.506080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hxx\" (UniqueName: \"kubernetes.io/projected/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-kube-api-access-x7hxx\") pod \"must-gather-z4j7h\" (UID: \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\") " pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:38:41.506351 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.506321    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-must-gather-output\") pod \"must-gather-z4j7h\" (UID: \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\") " pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:38:41.514964 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.514936    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hxx\" (UniqueName: \"kubernetes.io/projected/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-kube-api-access-x7hxx\") pod \"must-gather-z4j7h\" (UID: \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\") " pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:38:41.716329 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.716297    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:38:41.988314 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.988288    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnx75/must-gather-z4j7h"]
Sep 16 19:38:41.990960 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:38:41.990932    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87ecaa35_c0e7_4241_926b_a00c9c9fe72b.slice/crio-d91f6f1c56d450d9b22cba1791dde4773408d380a2dde90d4e6091d42c1265ed WatchSource:0}: Error finding container d91f6f1c56d450d9b22cba1791dde4773408d380a2dde90d4e6091d42c1265ed: Status 404 returned error can't find the container with id d91f6f1c56d450d9b22cba1791dde4773408d380a2dde90d4e6091d42c1265ed
Sep 16 19:38:41.992752 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:41.992735    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:38:42.449816 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:42.449778    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnx75/must-gather-z4j7h" event={"ID":"87ecaa35-c0e7-4241-926b-a00c9c9fe72b","Type":"ContainerStarted","Data":"d91f6f1c56d450d9b22cba1791dde4773408d380a2dde90d4e6091d42c1265ed"}
Sep 16 19:38:44.456310 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:44.456261    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 16 19:38:46.478865 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:46.478813    2576 generic.go:356] "Generic (PLEG): container finished" podID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerID="b8059629b14ae79dad1f1eb9477036662b0909449b67deec123a352d8f73a996" exitCode=0
Sep 16 19:38:46.479299 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:46.478892    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" event={"ID":"bc735b32-c4ab-4ec4-aaf4-b51852684b5e","Type":"ContainerDied","Data":"b8059629b14ae79dad1f1eb9477036662b0909449b67deec123a352d8f73a996"}
Sep 16 19:38:48.002672 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.002642    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:38:48.069189 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.069156    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-proxy-tls\") pod \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") "
Sep 16 19:38:48.069358 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.069207    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fghh\" (UniqueName: \"kubernetes.io/projected/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-api-access-7fghh\") pod \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") "
Sep 16 19:38:48.069358 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.069246    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-rbac-proxy-sar-config\") pod \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\" (UID: \"bc735b32-c4ab-4ec4-aaf4-b51852684b5e\") "
Sep 16 19:38:48.069641 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.069614    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc735b32-c4ab-4ec4-aaf4-b51852684b5e" (UID: "bc735b32-c4ab-4ec4-aaf4-b51852684b5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:38:48.071393 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.071357    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc735b32-c4ab-4ec4-aaf4-b51852684b5e" (UID: "bc735b32-c4ab-4ec4-aaf4-b51852684b5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:48.071518 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.071399    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-api-access-7fghh" (OuterVolumeSpecName: "kube-api-access-7fghh") pod "bc735b32-c4ab-4ec4-aaf4-b51852684b5e" (UID: "bc735b32-c4ab-4ec4-aaf4-b51852684b5e"). InnerVolumeSpecName "kube-api-access-7fghh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:48.170513 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.170474    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.170513 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.170513    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-proxy-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.170755 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.170529    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fghh\" (UniqueName: \"kubernetes.io/projected/bc735b32-c4ab-4ec4-aaf4-b51852684b5e-kube-api-access-7fghh\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.493262 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.493177    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"
Sep 16 19:38:48.493262 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.493176    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8" event={"ID":"bc735b32-c4ab-4ec4-aaf4-b51852684b5e","Type":"ContainerDied","Data":"24dd9bfdf9e6da8184780b0047c370e9ddc3e9451da9e4fc525a935b65762ffc"}
Sep 16 19:38:48.493564 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.493325    2576 scope.go:117] "RemoveContainer" containerID="99e46099dca9013dc85798daacf37cb30687e60dc751f597c9d69ab20ec7eb0a"
Sep 16 19:38:48.495601 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.495520    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnx75/must-gather-z4j7h" event={"ID":"87ecaa35-c0e7-4241-926b-a00c9c9fe72b","Type":"ContainerStarted","Data":"fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca"}
Sep 16 19:38:48.495601 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.495568    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnx75/must-gather-z4j7h" event={"ID":"87ecaa35-c0e7-4241-926b-a00c9c9fe72b","Type":"ContainerStarted","Data":"18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6"}
Sep 16 19:38:48.502655 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.502627    2576 scope.go:117] "RemoveContainer" containerID="b8059629b14ae79dad1f1eb9477036662b0909449b67deec123a352d8f73a996"
Sep 16 19:38:48.514479 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.514425    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bnx75/must-gather-z4j7h" podStartSLOduration=1.408273323 podStartE2EDuration="7.514410329s" totalImagesPullingTime="6.106137006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:48.513379021 +0000 UTC m=+3278.707242694" watchObservedRunningTime="2026-09-16 19:38:48.514410329 +0000 UTC m=+3278.708274000"
Sep 16 19:38:48.528009 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.527943    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"]
Sep 16 19:38:48.529585 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:48.529557    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0263c-predictor-5b95f74bf6-8v5n8"]
Sep 16 19:38:50.337226 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:38:50.337193    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" path="/var/lib/kubelet/pods/bc735b32-c4ab-4ec4-aaf4-b51852684b5e/volumes"
Sep 16 19:39:10.507654 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:39:10.505569    2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87ecaa35_c0e7_4241_926b_a00c9c9fe72b.slice/crio-18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87ecaa35_c0e7_4241_926b_a00c9c9fe72b.slice/crio-conmon-18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:39:10.581643 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:10.581527    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:39:10.589460 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:10.589435    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:39:10.613256 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:10.613222    2576 generic.go:356] "Generic (PLEG): container finished" podID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" containerID="18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6" exitCode=0
Sep 16 19:39:10.613429 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:10.613293    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnx75/must-gather-z4j7h" event={"ID":"87ecaa35-c0e7-4241-926b-a00c9c9fe72b","Type":"ContainerDied","Data":"18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6"}
Sep 16 19:39:10.613637 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:10.613624    2576 scope.go:117] "RemoveContainer" containerID="18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6"
Sep 16 19:39:11.495970 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:11.495938    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bnx75_must-gather-z4j7h_87ecaa35-c0e7-4241-926b-a00c9c9fe72b/gather/0.log"
Sep 16 19:39:14.829594 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:14.829562    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-49tdd_eb33c8b3-cc3f-45e3-a4a6-f154e5183033/global-pull-secret-syncer/0.log"
Sep 16 19:39:15.027025 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:15.026992    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hs9zv_64f66d84-84f6-4be7-892c-7163d7be2560/konnectivity-agent/0.log"
Sep 16 19:39:15.090242 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:15.090158    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-134.ec2.internal_4b26e3b59ca8b326d0e3c11878453793/haproxy/0.log"
Sep 16 19:39:16.978625 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:16.978586    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bnx75/must-gather-z4j7h"]
Sep 16 19:39:16.979044 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:16.978818    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-bnx75/must-gather-z4j7h" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" containerName="copy" containerID="cri-o://fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca" gracePeriod=2
Sep 16 19:39:16.980531 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:16.980504    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bnx75/must-gather-z4j7h"]
Sep 16 19:39:16.980813 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:39:16.980781    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z4j7h\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bnx75\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:39:17.216770 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.216739    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bnx75_must-gather-z4j7h_87ecaa35-c0e7-4241-926b-a00c9c9fe72b/copy/0.log"
Sep 16 19:39:17.219476 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.219454    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:39:17.221508 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:39:17.221479    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z4j7h\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bnx75\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:39:17.330428 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.330334    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7hxx\" (UniqueName: \"kubernetes.io/projected/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-kube-api-access-x7hxx\") pod \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\" (UID: \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\") "
Sep 16 19:39:17.330428 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.330373    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-must-gather-output\") pod \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\" (UID: \"87ecaa35-c0e7-4241-926b-a00c9c9fe72b\") "
Sep 16 19:39:17.331902 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.331870    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "87ecaa35-c0e7-4241-926b-a00c9c9fe72b" (UID: "87ecaa35-c0e7-4241-926b-a00c9c9fe72b"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:17.332590 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.332562    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-kube-api-access-x7hxx" (OuterVolumeSpecName: "kube-api-access-x7hxx") pod "87ecaa35-c0e7-4241-926b-a00c9c9fe72b" (UID: "87ecaa35-c0e7-4241-926b-a00c9c9fe72b"). InnerVolumeSpecName "kube-api-access-x7hxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:39:17.431203 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.431173    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7hxx\" (UniqueName: \"kubernetes.io/projected/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-kube-api-access-x7hxx\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:39:17.431203 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.431198    2576 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/87ecaa35-c0e7-4241-926b-a00c9c9fe72b-must-gather-output\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\""
Sep 16 19:39:17.646925 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.646831    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bnx75_must-gather-z4j7h_87ecaa35-c0e7-4241-926b-a00c9c9fe72b/copy/0.log"
Sep 16 19:39:17.647281 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.647260    2576 generic.go:356] "Generic (PLEG): container finished" podID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" containerID="fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca" exitCode=143
Sep 16 19:39:17.647373 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.647316    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:39:17.647415 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.647381    2576 scope.go:117] "RemoveContainer" containerID="fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca"
Sep 16 19:39:17.649325 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:39:17.649294    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z4j7h\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bnx75\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:39:17.655895 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.655875    2576 scope.go:117] "RemoveContainer" containerID="18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6"
Sep 16 19:39:17.657381 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:39:17.657356    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z4j7h\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bnx75\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" pod="openshift-must-gather-bnx75/must-gather-z4j7h"
Sep 16 19:39:17.667735 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.667711    2576 scope.go:117] "RemoveContainer" containerID="fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca"
Sep 16 19:39:17.668082 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:39:17.668060    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca\": container with ID starting with fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca not found: ID does not exist" containerID="fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca"
Sep 16 19:39:17.668132 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.668093    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca"} err="failed to get container status \"fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca\": rpc error: code = NotFound desc = could not find container \"fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca\": container with ID starting with fd4951ca8ecc2fafd744a628c3d5cfaaaf948e56743f10e56e84b44ac19ef1ca not found: ID does not exist"
Sep 16 19:39:17.668132 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.668114    2576 scope.go:117] "RemoveContainer" containerID="18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6"
Sep 16 19:39:17.668367 ip-10-0-130-134 kubenswrapper[2576]: E0916 19:39:17.668353    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6\": container with ID starting with 18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6 not found: ID does not exist" containerID="18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6"
Sep 16 19:39:17.668404 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:17.668372    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6"} err="failed to get container status \"18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6\": rpc error: code = NotFound desc = could not find container \"18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6\": container with ID starting with 18d948bb75510abd43b11a481575cb7be9e359a24f92adc0da67afe24aa10ba6 not found: ID does not exist"
Sep 16 19:39:18.338680 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:18.338627    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" path="/var/lib/kubelet/pods/87ecaa35-c0e7-4241-926b-a00c9c9fe72b/volumes"
Sep 16 19:39:18.906800 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:18.906771    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eda1ff5a-a3d7-474c-98e1-60101b05ef6f/alertmanager/0.log"
Sep 16 19:39:18.936144 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:18.936115    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eda1ff5a-a3d7-474c-98e1-60101b05ef6f/config-reloader/0.log"
Sep 16 19:39:18.959294 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:18.959269    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eda1ff5a-a3d7-474c-98e1-60101b05ef6f/kube-rbac-proxy-web/0.log"
Sep 16 19:39:18.990540 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:18.990508    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eda1ff5a-a3d7-474c-98e1-60101b05ef6f/kube-rbac-proxy/0.log"
Sep 16 19:39:19.019238 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.019209    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eda1ff5a-a3d7-474c-98e1-60101b05ef6f/kube-rbac-proxy-metric/0.log"
Sep 16 19:39:19.051362 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.051334    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eda1ff5a-a3d7-474c-98e1-60101b05ef6f/prom-label-proxy/0.log"
Sep 16 19:39:19.079102 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.079078    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eda1ff5a-a3d7-474c-98e1-60101b05ef6f/init-config-reloader/0.log"
Sep 16 19:39:19.123442 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.123409    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-2kkqb_23c96a4f-7cf4-4d6e-85ad-ea75b7761fdc/cluster-monitoring-operator/0.log"
Sep 16 19:39:19.264430 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.264398    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6ccc9987d6-bhkws_4ae43d45-2d4a-489f-8ec1-39fc6cf45ffc/metrics-server/0.log"
Sep 16 19:39:19.314379 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.314349    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-mfnd7_ec7653b1-1d65-4e72-b3ee-001e7bfb1331/monitoring-plugin/0.log"
Sep 16 19:39:19.464408 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.464381    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9569k_9b5a216e-bf71-48a7-8781-f3aa53a97e3d/node-exporter/0.log"
Sep 16 19:39:19.491713 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.491681    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9569k_9b5a216e-bf71-48a7-8781-f3aa53a97e3d/kube-rbac-proxy/0.log"
Sep 16 19:39:19.524248 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.524169    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9569k_9b5a216e-bf71-48a7-8781-f3aa53a97e3d/init-textfile/0.log"
Sep 16 19:39:19.649436 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.649407    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-bgqdv_c2dc7049-3be1-4518-8d6b-e40208ec0c97/kube-rbac-proxy-main/0.log"
Sep 16 19:39:19.682754 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.682726    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-bgqdv_c2dc7049-3be1-4518-8d6b-e40208ec0c97/kube-rbac-proxy-self/0.log"
Sep 16 19:39:19.711233 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.711202    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-bgqdv_c2dc7049-3be1-4518-8d6b-e40208ec0c97/openshift-state-metrics/0.log"
Sep 16 19:39:19.994611 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:19.994571    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-n9lq5_8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f/prometheus-operator/0.log"
Sep 16 19:39:20.022889 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:20.022860    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-n9lq5_8c1da3a0-ac71-47d9-8eb0-fcf5ad7b394f/kube-rbac-proxy/0.log"
Sep 16 19:39:20.059954 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:20.059925    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-j5gsq_8b22bba5-6ae8-4bd9-9c3b-966cc2675de3/prometheus-operator-admission-webhook/0.log"
Sep 16 19:39:20.097380 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:20.097352    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-664f4c4c65-jfztk_2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d/telemeter-client/0.log"
Sep 16 19:39:20.126303 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:20.126277    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-664f4c4c65-jfztk_2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d/reload/0.log"
Sep 16 19:39:20.153915 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:20.153886    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-664f4c4c65-jfztk_2d1bb671-f6d3-4c3c-b197-08ceb53d0d8d/kube-rbac-proxy/0.log"
Sep 16 19:39:21.707544 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:21.707512    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-pkgnb_e101c2f1-e28b-4910-8d86-9d3265060b4c/networking-console-plugin/0.log"
Sep 16 19:39:22.754682 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:22.754652    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-zcgn2_b99fc008-2504-4554-a770-92b9160a3e36/download-server/0.log"
Sep 16 19:39:23.384965 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.384935    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-7tcbv_fb408ab8-669f-4ae8-9e41-39ddddcfcb69/volume-data-source-validator/0.log"
Sep 16 19:39:23.487006 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.486969    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"]
Sep 16 19:39:23.487357 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487344    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container"
Sep 16 19:39:23.487407 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487358    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container"
Sep 16 19:39:23.487407 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487369    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" containerName="copy"
Sep 16 19:39:23.487407 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487375    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" containerName="copy"
Sep 16 19:39:23.487407 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487386    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" containerName="gather"
Sep 16 19:39:23.487407 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487391    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" containerName="gather"
Sep 16 19:39:23.487407 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487399    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kube-rbac-proxy"
Sep 16 19:39:23.487407 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487404    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kube-rbac-proxy"
Sep 16 19:39:23.487623 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487463    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kube-rbac-proxy"
Sep 16 19:39:23.487623 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487472    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" containerName="gather"
Sep 16 19:39:23.487623 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487478    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ecaa35-c0e7-4241-926b-a00c9c9fe72b" containerName="copy"
Sep 16 19:39:23.487623 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.487485    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc735b32-c4ab-4ec4-aaf4-b51852684b5e" containerName="kserve-container"
Sep 16 19:39:23.490371 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.490354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.492629 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.492603    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9fjpm\"/\"kube-root-ca.crt\""
Sep 16 19:39:23.492629 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.492603    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9fjpm\"/\"openshift-service-ca.crt\""
Sep 16 19:39:23.493041 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.493026    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9fjpm\"/\"default-dockercfg-64szc\""
Sep 16 19:39:23.508242 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.508208    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"]
Sep 16 19:39:23.585321 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.585275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vzl\" (UniqueName: \"kubernetes.io/projected/d8496a88-5ac1-4346-91ee-8962e0395e6e-kube-api-access-54vzl\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.585534 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.585331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-proc\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.585534 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.585363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-podres\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.585534 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.585433    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-sys\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.585534 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.585462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-lib-modules\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.685929 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.685886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-sys\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.685929 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.685934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-lib-modules\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.686165 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.685970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vzl\" (UniqueName: \"kubernetes.io/projected/d8496a88-5ac1-4346-91ee-8962e0395e6e-kube-api-access-54vzl\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.686165 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.685997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-proc\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.686165 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.686016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-podres\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.686165 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.686026    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-sys\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.686165 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.686088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-proc\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.686165 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.686160    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-podres\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.686358 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.686161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8496a88-5ac1-4346-91ee-8962e0395e6e-lib-modules\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.693328 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.693302    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vzl\" (UniqueName: \"kubernetes.io/projected/d8496a88-5ac1-4346-91ee-8962e0395e6e-kube-api-access-54vzl\") pod \"perf-node-gather-daemonset-hl6vw\" (UID: \"d8496a88-5ac1-4346-91ee-8962e0395e6e\") " pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:23.801598 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:23.801551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:24.076951 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:24.076927    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"]
Sep 16 19:39:24.079718 ip-10-0-130-134 kubenswrapper[2576]: W0916 19:39:24.079685    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd8496a88_5ac1_4346_91ee_8962e0395e6e.slice/crio-0ff02b34ec46a00117e1b0133d220f7bebd7a4ce1810910bc29d1c5d56fa09bf WatchSource:0}: Error finding container 0ff02b34ec46a00117e1b0133d220f7bebd7a4ce1810910bc29d1c5d56fa09bf: Status 404 returned error can't find the container with id 0ff02b34ec46a00117e1b0133d220f7bebd7a4ce1810910bc29d1c5d56fa09bf
Sep 16 19:39:24.329602 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:24.329516    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kv4z5_cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc/dns/0.log"
Sep 16 19:39:24.355527 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:24.355497    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kv4z5_cbc7ecbe-3313-46a8-a6cf-efb1c6b260fc/kube-rbac-proxy/0.log"
Sep 16 19:39:24.494426 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:24.494396    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6rmzs_641e581f-3294-4bfa-852e-8f3bc14f54df/dns-node-resolver/0.log"
Sep 16 19:39:24.682287 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:24.682254    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw" event={"ID":"d8496a88-5ac1-4346-91ee-8962e0395e6e","Type":"ContainerStarted","Data":"31c3a5a7b45089dd84639912382a6ce09604fb1db61cb1918d1c396de89b04fc"}
Sep 16 19:39:24.682478 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:24.682304    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw" event={"ID":"d8496a88-5ac1-4346-91ee-8962e0395e6e","Type":"ContainerStarted","Data":"0ff02b34ec46a00117e1b0133d220f7bebd7a4ce1810910bc29d1c5d56fa09bf"}
Sep 16 19:39:24.682478 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:24.682373    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:24.697214 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:24.697166    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw" podStartSLOduration=1.697146472 podStartE2EDuration="1.697146472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:24.696478622 +0000 UTC m=+3314.890342296" watchObservedRunningTime="2026-09-16 19:39:24.697146472 +0000 UTC m=+3314.891010145"
Sep 16 19:39:25.125142 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:25.125062    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x4m4g_04f86fca-369a-49ce-8bd8-b734182ec3a5/node-ca/0.log"
Sep 16 19:39:26.287620 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:26.287591    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tz7mv_e68a197f-86c7-45e7-a856-4b5574581420/serve-healthcheck-canary/0.log"
Sep 16 19:39:26.665707 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:26.665675    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-k6nld_2c017d51-ac33-45e1-83a3-0c08e71053cb/insights-operator/0.log"
Sep 16 19:39:26.666182 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:26.666157    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-k6nld_2c017d51-ac33-45e1-83a3-0c08e71053cb/insights-operator/1.log"
Sep 16 19:39:26.767522 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:26.767492    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9kcjh_6a62d4ce-d4f6-4c7c-bdc7-6fa626388458/kube-rbac-proxy/0.log"
Sep 16 19:39:26.792173 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:26.792148    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9kcjh_6a62d4ce-d4f6-4c7c-bdc7-6fa626388458/exporter/0.log"
Sep 16 19:39:26.817475 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:26.817446    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9kcjh_6a62d4ce-d4f6-4c7c-bdc7-6fa626388458/extractor/0.log"
Sep 16 19:39:29.208163 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:29.208127    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-bn2h6_5a0fda36-e8fe-44c8-8865-bde5526ca6fb/manager/0.log"
Sep 16 19:39:29.431506 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:29.431474    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-b7tbx_40eb6d96-5a5a-4878-968c-866566f99a0a/s3-init/0.log"
Sep 16 19:39:29.452120 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:29.452090    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-b7tbx_40eb6d96-5a5a-4878-968c-866566f99a0a/download-hf-model/0.log"
Sep 16 19:39:30.696959 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:30.696930    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9fjpm/perf-node-gather-daemonset-hl6vw"
Sep 16 19:39:33.827460 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:33.827433    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-ft4np_ec42b10f-f217-4581-a844-135cfa4a1e53/migrator/0.log"
Sep 16 19:39:33.852742 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:33.852707    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-ft4np_ec42b10f-f217-4581-a844-135cfa4a1e53/graceful-termination/0.log"
Sep 16 19:39:35.633733 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:35.633706    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8fxx_543dcafa-704f-4d94-8a68-6e6ba60429ae/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:39:35.660665 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:35.660633    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8fxx_543dcafa-704f-4d94-8a68-6e6ba60429ae/egress-router-binary-copy/0.log"
Sep 16 19:39:35.688426 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:35.688392    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8fxx_543dcafa-704f-4d94-8a68-6e6ba60429ae/cni-plugins/0.log"
Sep 16 19:39:35.716523 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:35.716493    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8fxx_543dcafa-704f-4d94-8a68-6e6ba60429ae/bond-cni-plugin/0.log"
Sep 16 19:39:35.746386 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:35.746360    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8fxx_543dcafa-704f-4d94-8a68-6e6ba60429ae/routeoverride-cni/0.log"
Sep 16 19:39:35.774222 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:35.774197    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8fxx_543dcafa-704f-4d94-8a68-6e6ba60429ae/whereabouts-cni-bincopy/0.log"
Sep 16 19:39:35.803845 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:35.803797    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8fxx_543dcafa-704f-4d94-8a68-6e6ba60429ae/whereabouts-cni/0.log"
Sep 16 19:39:35.922393 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:35.922363    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gckt2_ccbaaf68-77fa-4845-a85d-92b44a50a8ab/kube-multus/0.log"
Sep 16 19:39:36.002125 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:36.002087    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fzv4f_5d7a2ef5-a033-4dd2-89e3-bc86658a3373/network-metrics-daemon/0.log"
Sep 16 19:39:36.025583 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:36.025555    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fzv4f_5d7a2ef5-a033-4dd2-89e3-bc86658a3373/kube-rbac-proxy/0.log"
Sep 16 19:39:37.114500 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:37.114465    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-controller/0.log"
Sep 16 19:39:37.136938 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:37.136907    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/1.log"
Sep 16 19:39:37.137100 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:37.136993    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovn-acl-logging/0.log"
Sep 16 19:39:37.164684 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:37.164652    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/kube-rbac-proxy-node/0.log"
Sep 16 19:39:37.198261 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:37.198229    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:39:37.217506 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:37.217481    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/northd/0.log"
Sep 16 19:39:37.246963 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:37.246931    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/nbdb/0.log"
Sep 16 19:39:37.274692 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:37.274665    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/sbdb/0.log"
Sep 16 19:39:37.320135 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:37.320102    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4x5hs_c18ed6a5-1b44-4261-a9cc-c5e78164115e/ovnkube-controller/0.log"
Sep 16 19:39:38.698302 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:38.698270    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-422rl_243e8f33-96bd-4cbc-adae-88da4d776423/check-endpoints/0.log"
Sep 16 19:39:38.754617 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:38.754580    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-gglxk_234b9608-1129-44b3-9803-cdedb1a0d3f8/network-check-target-container/0.log"
Sep 16 19:39:39.930271 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:39.930245    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-x4vvb_4e8ec2ab-0c2f-4fff-83c4-a260a24907cc/iptables-alerter/0.log"
Sep 16 19:39:40.686594 ip-10-0-130-134 kubenswrapper[2576]: I0916 19:39:40.686563    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-lx59p_8a9e3f2d-f231-4b9c-be15-f6c370260974/tuned/0.log"