Jul 02 18:28:34.258443 ip-10-0-130-23 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 02 18:28:34.258458 ip-10-0-130-23 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 02 18:28:34.258468 ip-10-0-130-23 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 02 18:28:34.258773 ip-10-0-130-23 systemd[1]: Failed to start Kubernetes Kubelet. Jul 02 18:28:44.324113 ip-10-0-130-23 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 02 18:28:44.324132 ip-10-0-130-23 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d15ffd9223734a8a900e888016d9fbbc -- Jul 02 18:30:41.241660 ip-10-0-130-23 systemd[1]: Starting Kubernetes Kubelet... Jul 02 18:30:42.987880 ip-10-0-130-23 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 18:30:42.987880 ip-10-0-130-23 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 18:30:42.987880 ip-10-0-130-23 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 18:30:42.987880 ip-10-0-130-23 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 18:30:42.987880 ip-10-0-130-23 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 18:30:42.989505 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:42.989415 2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 18:30:42.997545 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997517 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 18:30:42.997545 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997537 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 18:30:42.997545 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997541 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 18:30:42.997545 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997544 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 18:30:42.997545 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997547 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 18:30:42.997545 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997550 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 18:30:42.997545 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997552 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 18:30:42.997545 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997555 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997560 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997564 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997567 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997570 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997572 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997575 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997578 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997580 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997583 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997585 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997589 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997593 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997596 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997599 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997602 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997605 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997608 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997610 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 18:30:42.997859 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997613 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997615 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997618 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997621 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997623 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997630 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997645 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997648 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997651 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997653 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997656 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997658 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997661 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997664 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997667 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997670 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997673 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997676 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997679 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997682 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 18:30:42.998310 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997684 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997687 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997690 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997692 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997695 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997698 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997700 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997703 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997705 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997708 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997710 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997713 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997715 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997718 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997720 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997723 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997725 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997728 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997730 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997733 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 18:30:42.998835 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997735 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997738 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997740 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997743 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997745 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997748 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997751 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997754 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997757 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997760 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997762 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997765 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997767 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997770 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997772 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997775 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997777 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997781 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997784 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997787 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 18:30:42.999342 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997789 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997791 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997796 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997799 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997802 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997805 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997808 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997810 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997814 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997816 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997820 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997830 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.997833 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999518 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999525 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999529 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999532 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999536 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999540 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 18:30:42.999841 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999544 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999547 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999550 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999552 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999555 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999558 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999560 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999563 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999565 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999568 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999570 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999573 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999576 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999579 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999581 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999585 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999588 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999590 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999593 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999596 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 18:30:43.000321 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999599 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999601 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999604 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999607 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999611 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999614 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999616 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999619 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999622 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999625 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999628 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999631 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999647 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999650 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999652 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999655 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999657 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999660 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999663 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 18:30:43.000828 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999666 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999668 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999671 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999673 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999676 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999679 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999682 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999684 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999687 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999690 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999692 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999695 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999698 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999700 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999702 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999705 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999708 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999711 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999713 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999716 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 18:30:43.001324 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999725 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999728 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999731 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999736 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999738 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999741 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999744 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999746 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999749 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999751 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999754 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999756 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999760 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999764 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999767 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999769 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999772 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999775 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999778 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999781 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 18:30:43.001822 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999784 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999787 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999789 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999792 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999794 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999797 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999800 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999802 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999805 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999808 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999810 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999813 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999815 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:42.999818 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000201 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000206 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000209 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000212 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000214 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000217 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 18:30:43.002313 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000219 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000221 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000224 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000226 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000229 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000232 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000235 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000238 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000241 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000243 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000246 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000248 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000251 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000253 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000256 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000258 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000261 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000263 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000265 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000268 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 18:30:43.002816 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000270 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000272 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000275 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000278 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000281 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000284 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000286 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000289 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000292 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000294 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000297 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000299 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000301 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000304 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000306 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000309 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000313 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000316 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000318 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 18:30:43.003298 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000321 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000326 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000330 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000333 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000336 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000339 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000341 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000344 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000347 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000349 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000352 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000354 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000357 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000359 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000361 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000364 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000366 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000369 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000371 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000374 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 18:30:43.003790 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000377 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000380 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000382 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000385 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000387 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000391 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000393 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000396 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000398 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000401 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000403 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000406 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000408 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000411 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000414 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000416 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000418 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000421 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000424 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000426 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 18:30:43.004261 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000428 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000431 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000433 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000436 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000438 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000440 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000443 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000445 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000448 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000450 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000453 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000455 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000461 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.000463 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000532 2552 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000540 2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000547 2552 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000554 2552 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000560 2552 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000565 2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000571 2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 18:30:43.004759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000576 2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000579 2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000582 2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000585 2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000589 2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000592 2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000595 2552 flags.go:64] FLAG: --cgroup-root="" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000598 2552 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000601 2552 flags.go:64] FLAG: --client-ca-file="" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000604 2552 flags.go:64] FLAG: --cloud-provider="external" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000608 2552 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000612 2552 flags.go:64] FLAG: --cluster-domain="" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000615 2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000618 2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000621 2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000624 2552 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000628 2552 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000643 2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000646 2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000649 2552 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000652 2552 flags.go:64] FLAG: --contention-profiling="false" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000655 2552 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000658 2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000664 2552 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 18:30:43.005262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000666 2552 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000671 2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000674 2552 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000677 2552 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000680 2552 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000683 2552 flags.go:64] FLAG: --enable-server="true" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000686 2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000691 2552 flags.go:64] FLAG: --event-burst="100" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000694 2552 flags.go:64] FLAG: --event-qps="50" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000697 2552 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000700 2552 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000704 2552 flags.go:64] FLAG: --eviction-hard="" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000708 2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000711 2552 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000714 2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000717 2552 flags.go:64] FLAG: --eviction-soft="" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000720 2552 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000723 2552 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000726 2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000729 2552 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000732 2552 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000735 2552 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000738 2552 flags.go:64] FLAG: --feature-gates="" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000741 2552 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000744 2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 18:30:43.005846 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000747 2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000750 2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000753 2552 flags.go:64] FLAG: --healthz-port="10248" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000756 2552 flags.go:64] FLAG: --help="false" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000759 2552 flags.go:64] FLAG: --hostname-override="ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000762 2552 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000766 2552 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000769 2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000772 2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000775 2552 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000778 2552 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000781 2552 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000783 2552 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000786 2552 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000789 2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000792 2552 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000795 2552 flags.go:64] FLAG: --kube-reserved="" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000798 2552 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000802 2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000805 2552 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000808 2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000811 2552 flags.go:64] FLAG: --lock-file="" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000814 2552 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000816 2552 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 18:30:43.006507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000820 2552 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000825 2552 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000829 2552 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000832 2552 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000835 2552 flags.go:64] FLAG: --logging-format="text" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000838 2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000842 2552 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000845 2552 flags.go:64] FLAG: --manifest-url="" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000848 2552 flags.go:64] FLAG: --manifest-url-header="" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000852 2552 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000855 2552 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000859 2552 flags.go:64] FLAG: --max-pods="110" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000863 2552 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000866 2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000869 2552 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000872 2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000875 2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000878 2552 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000881 2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000888 2552 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000891 2552 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000894 2552 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000897 2552 flags.go:64] FLAG: --pod-cidr="" Jul 02 18:30:43.007150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000900 2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000906 2552 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000909 2552 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000911 2552 flags.go:64] FLAG: --pods-per-core="0" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000916 2552 flags.go:64] FLAG: --port="10250" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000919 2552 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000922 2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0607a559616874a21" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000925 2552 flags.go:64] FLAG: --qos-reserved="" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000928 2552 flags.go:64] FLAG: --read-only-port="10255" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000931 2552 flags.go:64] FLAG: --register-node="true" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000934 2552 flags.go:64] FLAG: --register-with-taints="" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000937 2552 flags.go:64] FLAG: --registry-burst="10" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000941 2552 flags.go:64] FLAG: --registry-qps="5" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000943 2552 flags.go:64] FLAG: --reserved-cpus="" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000946 2552 flags.go:64] FLAG: --reserved-memory="" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000950 2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000953 2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000956 2552 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000958 2552 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000961 2552 flags.go:64] FLAG: --runonce="false" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000964 2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000967 2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000970 2552 flags.go:64] FLAG: --seccomp-default="false" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000973 2552 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000975 2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000978 2552 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 18:30:43.007778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000981 2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000985 2552 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000987 2552 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000990 2552 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000993 2552 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000996 2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.000999 2552 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001002 2552 flags.go:64] FLAG: --system-cgroups="" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001005 2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001010 2552 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001015 2552 flags.go:64] FLAG: --tls-cert-file="" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001017 2552 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001022 2552 flags.go:64] FLAG: --tls-min-version="" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001024 2552 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001027 2552 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001030 2552 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001033 2552 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001036 2552 flags.go:64] FLAG: --v="2" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001040 2552 flags.go:64] FLAG: --version="false" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001045 2552 flags.go:64] FLAG: --vmodule="" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001048 2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001051 2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001174 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001178 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001181 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 18:30:43.008420 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001184 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001187 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001189 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001192 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001194 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001197 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001199 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001201 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001204 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001206 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001209 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001211 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001214 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001217 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001219 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001222 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001224 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001228 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001231 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001234 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 18:30:43.009041 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001236 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001240 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001242 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001245 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001247 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001250 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001252 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001255 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001258 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001260 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001263 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001265 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001268 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001271 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001273 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001276 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001278 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001281 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001283 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001286 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 18:30:43.009522 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001288 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001291 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001293 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001297 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001300 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001303 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001306 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001309 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001312 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001316 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001319 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001323 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001326 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001329 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001332 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001335 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001338 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001341 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001344 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 18:30:43.010065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001346 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001349 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001352 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001355 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001357 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001360 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001362 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001365 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001367 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001371 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001375 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001377 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001380 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001383 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001386 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001388 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001391 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001393 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001395 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 18:30:43.010531 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001398 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001400 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001403 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001407 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001410 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001414 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001417 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001420 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001423 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001427 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001429 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001432 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001435 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001437 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001440 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001443 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001446 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 18:30:43.011054 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.001448 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.001949 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.008760 2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.008775 2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.009630 2552 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.009658 2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009719 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009725 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009728 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009732 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009735 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009739 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009743 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009746 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 18:30:43.011468 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009750 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009753 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009756 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009759 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009762 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009764 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009767 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009769 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009772 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009775 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009777 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009780 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009783 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009785 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009788 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009791 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009794 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009797 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009799 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009802 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 18:30:43.011838 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009805 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009808 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009811 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009813 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009816 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009818 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009821 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009823 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009826 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009828 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009831 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009833 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009836 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009839 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009841 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009844 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009846 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009849 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009851 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009854 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 18:30:43.012304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009856 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009859 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009861 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009864 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009866 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009869 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009871 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009874 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009877 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009880 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009882 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009885 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009888 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009891 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009894 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009897 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009900 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009902 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009905 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 18:30:43.012800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009909 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009912 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009915 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009917 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009920 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009923 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009925 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009927 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009930 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009933 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009935 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009938 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009940 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009943 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009945 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009947 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009950 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009953 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009955 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009958 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 18:30:43.013262 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009960 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009963 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009965 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009968 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009971 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009974 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009977 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009980 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009982 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009985 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009988 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.009990 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.009995 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010112 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010117 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 18:30:43.013746 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010120 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010123 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010126 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010129 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010132 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010135 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010138 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010141 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010144 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010146 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010149 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010152 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010155 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010157 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010160 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010163 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010166 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010169 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010172 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010175 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 18:30:43.014155 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010178 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010181 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010184 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010187 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010190 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010192 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010194 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010197 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010200 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010202 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010205 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010207 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010210 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010212 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010215 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010217 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010220 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010222 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010225 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010229 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 18:30:43.014781 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010232 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010236 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010239 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010242 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010244 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010247 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010250 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010252 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010255 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010258 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010260 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010263 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010266 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010268 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010273 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010277 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010280 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010283 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010286 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 18:30:43.015296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010288 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010291 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010293 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010296 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010299 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010301 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010303 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010306 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010308 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010311 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010313 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010315 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010318 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010320 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010323 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010327 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010330 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010333 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010336 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010338 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 18:30:43.015860 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010341 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010344 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010346 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010349 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010351 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010354 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010356 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010359 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010362 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010364 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010367 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010369 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010372 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010374 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010377 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010379 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010382 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 18:30:43.016336 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:43.010384 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 18:30:43.016798 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.010389 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 18:30:43.016798 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.010489 2552 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 18:30:43.016798 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.013727 2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 18:30:43.016798 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.014888 2552 server.go:1019] "Starting client certificate rotation" Jul 02 18:30:43.016798 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.014987 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 18:30:43.016798 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.015020 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 18:30:43.017100 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.017086 2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 18:30:43.020111 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.020086 2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 18:30:43.031144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.031123 2552 log.go:25] "Validated CRI v1 runtime API" Jul 02 18:30:43.033170 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.033155 2552 log.go:25] "Validated CRI v1 image API" Jul 02 18:30:43.034759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.034742 2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 18:30:43.037256 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.037236 2552 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bacf1f44-b266-4383-a9a0-f943fa38bbe2:/dev/nvme0n1p3 cdfea8d7-fba4-4fec-a91f-26ad12139cdd:/dev/nvme0n1p4] Jul 02 18:30:43.037340 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.037256 2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 18:30:43.042336 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.042228 2552 manager.go:216] Machine: {Timestamp:2026-07-02 18:30:43.040916002 +0000 UTC m=+0.327265838 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099934 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f57ba826f613395e8766642275809 SystemUUID:ec2f57ba-826f-6133-95e8-766642275809 BootID:d15ffd92-2373-4a8a-900e-888016d9fbbc Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:08:cc:63:35:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:08:cc:63:35:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:87:34:a3:60:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 18:30:43.042336 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.042330 2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 18:30:43.042504 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.042422 2552 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 18:30:43.042555 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.042516 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 18:30:43.045102 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.045077 2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 18:30:43.045268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.045107 2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-23.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 18:30:43.045767 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.045756 2552 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 18:30:43.045830 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.045772 2552 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 18:30:43.045830 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.045783 2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 18:30:43.046324 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.046312 2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 18:30:43.047504 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.047492 2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 18:30:43.050648 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.050622 2552 state_mem.go:36] "Initialized new in-memory state store" Jul 02 18:30:43.050789 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.050778 2552 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 18:30:43.050844 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.050818 2552 kubelet.go:486] "Attempting to sync node with API server" Jul 02 18:30:43.050844 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.050834 2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 18:30:43.050938 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.050849 2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 18:30:43.050938 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.050863 2552 kubelet.go:398] "Adding apiserver pod source" Jul 02 18:30:43.050938 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.050875 2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 18:30:43.051276 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.051257 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 18:30:43.051338 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.051286 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 18:30:43.053559 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.053539 2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 18:30:43.055625 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.055607 2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 18:30:43.055715 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.055665 2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 18:30:43.056748 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056736 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 18:30:43.056793 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056752 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 18:30:43.056793 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056773 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 18:30:43.056793 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056778 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 18:30:43.056793 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056787 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 18:30:43.056793 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056792 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 18:30:43.056920 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056800 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 18:30:43.056920 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056805 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 18:30:43.056920 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056811 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 18:30:43.056920 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056817 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 18:30:43.056920 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056829 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 18:30:43.056920 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.056834 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 18:30:43.057541 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.057531 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 18:30:43.057541 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.057539 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 18:30:43.062035 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.061999 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 18:30:43.062035 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.062025 2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-23.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 18:30:43.062149 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.062069 2552 server.go:1268] "Started kubelet" Jul 02 18:30:43.062149 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.062071 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 18:30:43.062232 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.062146 2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 18:30:43.062232 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.062147 2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 18:30:43.062317 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.062261 2552 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 18:30:43.062928 ip-10-0-130-23 systemd[1]: Started Kubernetes Kubelet. Jul 02 18:30:43.063837 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.063821 2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 18:30:43.064270 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.064256 2552 server.go:310] "Adding debug handlers to kubelet server" Jul 02 18:30:43.067059 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.067037 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9j7ls" Jul 02 18:30:43.067577 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.066679 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-23.ec2.internal.18be8c8dc82d58a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-23.ec2.internal,UID:ip-10-0-130-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-23.ec2.internal,},FirstTimestamp:2026-07-02 18:30:43.061987492 +0000 UTC m=+0.348337326,LastTimestamp:2026-07-02 18:30:43.061987492 +0000 UTC m=+0.348337326,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-23.ec2.internal,}" Jul 02 18:30:43.067950 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.067930 2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 18:30:43.069723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.069710 2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 18:30:43.069778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.069726 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 18:30:43.069778 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.069772 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.070389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.070373 2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 18:30:43.070476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.070368 2552 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 18:30:43.070476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.070411 2552 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 18:30:43.070555 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.070508 2552 reconstruct.go:97] "Volume reconstruction finished" Jul 02 18:30:43.070555 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.070517 2552 reconciler.go:29] "Reconciler: start to sync state" Jul 02 18:30:43.070667 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.070565 2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 18:30:43.070667 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.070578 2552 factory.go:55] Registering systemd factory Jul 02 18:30:43.070667 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.070590 2552 factory.go:223] Registration of the systemd container factory successfully Jul 02 18:30:43.071085 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.071070 2552 factory.go:153] Registering CRI-O factory Jul 02 18:30:43.071170 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.071161 2552 factory.go:223] Registration of the crio container factory successfully Jul 02 18:30:43.071244 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.071238 2552 factory.go:103] Registering Raw factory Jul 02 18:30:43.071314 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.071307 2552 manager.go:1196] Started watching for new ooms in manager Jul 02 18:30:43.071859 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.071845 2552 manager.go:349] Starting recovery of all containers Jul 02 18:30:43.072226 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.072203 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 18:30:43.072302 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.072202 2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 18:30:43.075437 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.075416 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9j7ls" Jul 02 18:30:43.083765 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.083745 2552 manager.go:354] Recovery completed Jul 02 18:30:43.089074 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.089055 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 18:30:43.091622 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.091608 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientMemory" Jul 02 18:30:43.091712 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.091652 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 18:30:43.091712 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.091670 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientPID" Jul 02 18:30:43.092224 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.092209 2552 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 18:30:43.092224 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.092222 2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 18:30:43.092224 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.092225 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 18:30:43.092377 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.092236 2552 state_mem.go:36] "Initialized new in-memory state store" Jul 02 18:30:43.092900 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.092886 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientMemory" Jul 02 18:30:43.092973 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.092910 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 18:30:43.092973 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.092920 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientPID" Jul 02 18:30:43.093384 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.093373 2552 policy_none.go:49] "None policy: Start" Jul 02 18:30:43.093417 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.093389 2552 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 18:30:43.093417 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.093399 2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 18:30:43.095223 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.095213 2552 policy_none.go:47] "Start" Jul 02 18:30:43.137339 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.137195 2552 manager.go:337] "Starting Device Plugin manager" Jul 02 18:30:43.137454 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.137367 2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 18:30:43.137454 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.137380 2552 server.go:87] "Starting device plugin registration server" Jul 02 18:30:43.137702 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.137689 2552 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 18:30:43.137780 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.137704 2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 18:30:43.137845 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.137810 2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 18:30:43.137936 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.137924 2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 18:30:43.137986 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.137937 2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 18:30:43.138440 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.138422 2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 18:30:43.138527 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.138476 2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.228849 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.228771 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 18:30:43.230198 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.230037 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 18:30:43.230198 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.230069 2552 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 18:30:43.230198 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.230109 2552 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 18:30:43.230198 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.230144 2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 18:30:43.232282 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.232259 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 18:30:43.238130 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.238115 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 18:30:43.238891 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.238876 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientMemory" Jul 02 18:30:43.238970 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.238906 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 18:30:43.238970 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.238921 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientPID" Jul 02 18:30:43.238970 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.238943 2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.246446 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.246431 2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.246508 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.246452 2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-23.ec2.internal\": node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.262534 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.262515 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.331254 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.331227 2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal"] Jul 02 18:30:43.331361 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.331292 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 18:30:43.332757 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.332734 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientMemory" Jul 02 18:30:43.332866 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.332765 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 18:30:43.332866 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.332779 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientPID" Jul 02 18:30:43.334065 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.334050 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 18:30:43.334183 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.334155 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.334183 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.334181 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 18:30:43.334942 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.334926 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientMemory" Jul 02 18:30:43.335010 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.334953 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 18:30:43.335010 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.334963 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientPID" Jul 02 18:30:43.335010 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.334927 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientMemory" Jul 02 18:30:43.335110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.335030 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 18:30:43.335110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.335063 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientPID" Jul 02 18:30:43.336231 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.336218 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.336276 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.336249 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 18:30:43.336954 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.336941 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientMemory" Jul 02 18:30:43.337000 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.336964 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 18:30:43.337000 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.336973 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeHasSufficientPID" Jul 02 18:30:43.362756 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.362736 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.368241 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.368225 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-23.ec2.internal\" not found" node="ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.371567 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.371545 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/88c93248cc52546e186356ec011b8aa9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal\" (UID: \"88c93248cc52546e186356ec011b8aa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.371664 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.371575 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88c93248cc52546e186356ec011b8aa9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal\" (UID: \"88c93248cc52546e186356ec011b8aa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.371664 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.371651 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26e94d30503c9daf0c2ddc17f9303308-config\") pod \"kube-apiserver-proxy-ip-10-0-130-23.ec2.internal\" (UID: \"26e94d30503c9daf0c2ddc17f9303308\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.372797 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.372782 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-23.ec2.internal\" not found" node="ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.463738 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.463694 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.471979 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.471960 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/88c93248cc52546e186356ec011b8aa9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal\" (UID: \"88c93248cc52546e186356ec011b8aa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.472032 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.471985 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88c93248cc52546e186356ec011b8aa9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal\" (UID: \"88c93248cc52546e186356ec011b8aa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.472032 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.472002 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26e94d30503c9daf0c2ddc17f9303308-config\") pod \"kube-apiserver-proxy-ip-10-0-130-23.ec2.internal\" (UID: \"26e94d30503c9daf0c2ddc17f9303308\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.472032 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.472030 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26e94d30503c9daf0c2ddc17f9303308-config\") pod \"kube-apiserver-proxy-ip-10-0-130-23.ec2.internal\" (UID: \"26e94d30503c9daf0c2ddc17f9303308\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.472124 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.472054 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/88c93248cc52546e186356ec011b8aa9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal\" (UID: \"88c93248cc52546e186356ec011b8aa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.472124 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.472066 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88c93248cc52546e186356ec011b8aa9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal\" (UID: \"88c93248cc52546e186356ec011b8aa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.564462 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.564387 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.664911 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.664883 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.671129 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.671115 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.676131 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.676117 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.765924 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.765894 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.866517 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:43.866444 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-23.ec2.internal\" not found" Jul 02 18:30:43.888250 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.888231 2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 18:30:43.970156 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.970128 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.981602 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.981586 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 18:30:43.983099 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.983087 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal" Jul 02 18:30:43.991655 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:43.991627 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 18:30:44.014782 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.014762 2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 18:30:44.014911 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.014895 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 18:30:44.014958 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.014926 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 18:30:44.051010 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.050989 2552 apiserver.go:52] "Watching apiserver" Jul 02 18:30:44.064991 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.064967 2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 18:30:44.066101 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.066078 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-7x7hr","openshift-network-operator/iptables-alerter-jmq4v","openshift-ovn-kubernetes/ovnkube-node-59qz5","kube-system/global-pull-secret-syncer-hjqpc","kube-system/konnectivity-agent-qkjmb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k","openshift-cluster-node-tuning-operator/tuned-p4w84","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal","openshift-multus/multus-additional-cni-plugins-r52rg","openshift-multus/network-metrics-daemon-qc7bk","openshift-network-diagnostics/network-check-target-n98x8","kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal","openshift-dns/node-resolver-hcdmt","openshift-image-registry/node-ca-gh2rn"] Jul 02 18:30:44.068192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.068170 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.069677 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.069660 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.069866 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.069841 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 18:30:44.070015 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.069998 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 18:30:44.070093 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.070057 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 18:30:44.070093 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.070069 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 18:30:44.070194 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.070104 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 18:30:44.070194 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.070175 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dx2f4\"" Jul 02 18:30:44.070379 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.070368 2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 18:30:44.071048 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.071032 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.071447 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.071431 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 18:30:44.071521 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.071445 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 18:30:44.071521 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.071435 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 18:30:44.071701 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.071688 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kzbcl\"" Jul 02 18:30:44.072267 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.072255 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.072327 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.072310 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:30:44.072543 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.072529 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 18:30:44.072648 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.072609 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-td6xm\"" Jul 02 18:30:44.072725 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.072713 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 18:30:44.072942 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.072913 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 18:30:44.073010 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.072995 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 18:30:44.073060 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.073024 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 18:30:44.073331 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.073304 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 18:30:44.073730 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.073714 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:30:44.075051 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075016 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075051 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075044 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-system-cni-dir\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.075241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075068 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-var-lib-kubelet\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.075241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075092 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-run-ovn\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075114 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11a0b4e9-c404-4cf5-b70d-8736829d5821-env-overrides\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075138 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11a0b4e9-c404-4cf5-b70d-8736829d5821-ovn-node-metrics-cert\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075163 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qw78\" (UniqueName: \"kubernetes.io/projected/11a0b4e9-c404-4cf5-b70d-8736829d5821-kube-api-access-6qw78\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075185 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc54c34b-e67c-4e6a-bd82-a00e09e2d94e-host-slash\") pod \"iptables-alerter-jmq4v\" (UID: \"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e\") " pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.075241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075207 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-cni-dir\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.075241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075232 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-cni-binary-copy\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075253 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-run-multus-certs\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075275 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-systemd-units\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075298 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-cni-bin\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075321 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11a0b4e9-c404-4cf5-b70d-8736829d5821-ovnkube-config\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075333 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075341 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-hostroot\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075354 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-68526\"" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075363 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-kubelet\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075385 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-run-systemd\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075402 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075417 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-etc-openvswitch\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075445 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-run-ovn-kubernetes\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075467 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-var-lib-cni-bin\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075489 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-conf-dir\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075512 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-slash\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075537 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-log-socket\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075420 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075586 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-cni-netd\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.075723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075627 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-cnibin\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075674 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-socket-dir-parent\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075712 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-run-netns\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075753 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-daemon-config\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075775 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-etc-kubernetes\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075812 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwlht\" (UniqueName: \"kubernetes.io/projected/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-kube-api-access-lwlht\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075841 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc54c34b-e67c-4e6a-bd82-a00e09e2d94e-iptables-alerter-script\") pod \"iptables-alerter-jmq4v\" (UID: \"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e\") " pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075865 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862sx\" (UniqueName: \"kubernetes.io/projected/cc54c34b-e67c-4e6a-bd82-a00e09e2d94e-kube-api-access-862sx\") pod \"iptables-alerter-jmq4v\" (UID: \"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e\") " pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075886 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-os-release\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075909 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-var-lib-openvswitch\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075937 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-run-openvswitch\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.075976 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-node-log\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.076002 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11a0b4e9-c404-4cf5-b70d-8736829d5821-ovnkube-script-lib\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.076032 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-run-k8s-cni-cncf-io\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.076066 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-var-lib-cni-multus\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.076386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.076094 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-run-netns\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.076875 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.076806 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 18:30:44.077030 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.077014 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 18:30:44.077105 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.077034 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jb8dr\"" Jul 02 18:30:44.077105 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.077089 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.077229 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.077214 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 18:30:44.078182 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.078167 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 18:30:44.078516 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.078502 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 18:30:44.078813 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.078797 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-644jm\"" Jul 02 18:30:44.078813 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.078810 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 18:30:44.078992 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.078976 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.079797 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.079761 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 18:25:43 +0000 UTC" deadline="2028-04-11 02:33:20.358757328 +0000 UTC" Jul 02 18:30:44.079848 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.079797 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15560h2m36.278962625s" Jul 02 18:30:44.080285 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.080274 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:44.080343 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.080324 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:30:44.080726 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.080711 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 18:30:44.080802 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.080724 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 18:30:44.080802 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.080724 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2vtln\"" Jul 02 18:30:44.081434 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.081419 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:44.081485 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.081471 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:30:44.083460 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.083010 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.085066 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.084750 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 18:30:44.085066 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.084800 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.085066 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.085023 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 18:30:44.085237 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.085203 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j4qgk\"" Jul 02 18:30:44.086319 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.086301 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 18:30:44.086942 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.086927 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 18:30:44.087043 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.087026 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-26cnz\"" Jul 02 18:30:44.087086 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.087072 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 18:30:44.103527 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.103505 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r84zh" Jul 02 18:30:44.110931 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.110913 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r84zh" Jul 02 18:30:44.171399 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.171383 2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 18:30:44.177142 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177117 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-sys\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.177237 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177160 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-hostroot\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177237 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177186 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-run-systemd\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177237 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177207 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-host\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.177237 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177228 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-tuned\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177240 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-hostroot\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177250 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4de0e2a-fa13-4ffe-8835-9545f4f925af-tmp\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177286 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-var-lib-cni-bin\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177312 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-slash\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177331 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-log-socket\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177309 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-run-systemd\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177354 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-cni-netd\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177371 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-slash\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177378 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-socket-dir-parent\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177410 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177392 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-log-socket\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177416 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-var-lib-cni-bin\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177416 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-run-netns\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177443 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-daemon-config\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177459 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc54c34b-e67c-4e6a-bd82-a00e09e2d94e-iptables-alerter-script\") pod \"iptables-alerter-jmq4v\" (UID: \"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e\") " pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177458 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-cni-netd\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177474 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862sx\" (UniqueName: \"kubernetes.io/projected/cc54c34b-e67c-4e6a-bd82-a00e09e2d94e-kube-api-access-862sx\") pod \"iptables-alerter-jmq4v\" (UID: \"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e\") " pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177493 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-var-lib-openvswitch\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177498 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-socket-dir-parent\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177533 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-run-netns\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177601 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-node-log\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177646 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqnt\" (UniqueName: \"kubernetes.io/projected/7560ae93-bfc6-4b22-acde-054add93fe8d-kube-api-access-ltqnt\") pod \"node-resolver-hcdmt\" (UID: \"7560ae93-bfc6-4b22-acde-054add93fe8d\") " pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177668 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-var-lib-openvswitch\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177675 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-os-release\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177692 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-node-log\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177704 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d103adac-ed3a-485a-90c7-9713adec7aba-dbus\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177730 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.177883 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177748 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3016b638-7d5e-49c8-9a84-034ff4ab17f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177750 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-os-release\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177764 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9f69688-0465-473e-bcab-909e7800ee07-host\") pod \"node-ca-gh2rn\" (UID: \"c9f69688-0465-473e-bcab-909e7800ee07\") " pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177814 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-run-k8s-cni-cncf-io\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177835 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-run-netns\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177854 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177876 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-sysctl-d\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177888 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-run-k8s-cni-cncf-io\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177903 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-system-cni-dir\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177918 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-run-netns\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177923 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11a0b4e9-c404-4cf5-b70d-8736829d5821-ovn-node-metrics-cert\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177960 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qw78\" (UniqueName: \"kubernetes.io/projected/11a0b4e9-c404-4cf5-b70d-8736829d5821-kube-api-access-6qw78\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177963 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-system-cni-dir\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.177987 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d68dc90b-8983-41f7-a1d7-6c23581adfbe-agent-certs\") pod \"konnectivity-agent-qkjmb\" (UID: \"d68dc90b-8983-41f7-a1d7-6c23581adfbe\") " pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178011 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-socket-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178011 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc54c34b-e67c-4e6a-bd82-a00e09e2d94e-iptables-alerter-script\") pod \"iptables-alerter-jmq4v\" (UID: \"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e\") " pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178043 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-daemon-config\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.178696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178048 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-device-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178097 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc54c34b-e67c-4e6a-bd82-a00e09e2d94e-host-slash\") pod \"iptables-alerter-jmq4v\" (UID: \"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e\") " pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178125 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-var-lib-kubelet\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178153 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngn2\" (UniqueName: \"kubernetes.io/projected/c4de0e2a-fa13-4ffe-8835-9545f4f925af-kube-api-access-cngn2\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178162 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc54c34b-e67c-4e6a-bd82-a00e09e2d94e-host-slash\") pod \"iptables-alerter-jmq4v\" (UID: \"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e\") " pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178180 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-cnibin\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178219 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-cni-binary-copy\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178232 2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178255 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3016b638-7d5e-49c8-9a84-034ff4ab17f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178278 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9f69688-0465-473e-bcab-909e7800ee07-serviceca\") pod \"node-ca-gh2rn\" (UID: \"c9f69688-0465-473e-bcab-909e7800ee07\") " pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178300 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2njn5\" (UniqueName: \"kubernetes.io/projected/c9f69688-0465-473e-bcab-909e7800ee07-kube-api-access-2njn5\") pod \"node-ca-gh2rn\" (UID: \"c9f69688-0465-473e-bcab-909e7800ee07\") " pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178322 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-kubelet\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178366 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-etc-openvswitch\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178397 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-run-ovn-kubernetes\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178416 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-kubelet\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178424 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3016b638-7d5e-49c8-9a84-034ff4ab17f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178456 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-conf-dir\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.179321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178481 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d68dc90b-8983-41f7-a1d7-6c23581adfbe-konnectivity-ca\") pod \"konnectivity-agent-qkjmb\" (UID: \"d68dc90b-8983-41f7-a1d7-6c23581adfbe\") " pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178485 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-run-ovn-kubernetes\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178504 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7560ae93-bfc6-4b22-acde-054add93fe8d-hosts-file\") pod \"node-resolver-hcdmt\" (UID: \"7560ae93-bfc6-4b22-acde-054add93fe8d\") " pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178504 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-conf-dir\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178455 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-etc-openvswitch\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178556 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-etc-selinux\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178583 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-sysconfig\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178610 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-cnibin\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178650 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-cni-binary-copy\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178673 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-etc-kubernetes\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178701 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwlht\" (UniqueName: \"kubernetes.io/projected/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-kube-api-access-lwlht\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178702 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-etc-kubernetes\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178679 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-cnibin\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178721 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-run-openvswitch\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178787 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-systemd\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178810 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-lib-modules\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178745 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-run-openvswitch\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.180144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178833 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j68lk\" (UniqueName: \"kubernetes.io/projected/3016b638-7d5e-49c8-9a84-034ff4ab17f7-kube-api-access-j68lk\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178882 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11a0b4e9-c404-4cf5-b70d-8736829d5821-ovnkube-script-lib\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178920 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-modprobe-d\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178956 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-sysctl-conf\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.178980 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-system-cni-dir\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179006 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179032 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-var-lib-cni-multus\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179057 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179085 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tlv6\" (UniqueName: \"kubernetes.io/projected/d1038ec5-e712-43d1-bf5e-8370bcefd309-kube-api-access-8tlv6\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179108 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d103adac-ed3a-485a-90c7-9713adec7aba-kubelet-config\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179122 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-var-lib-cni-multus\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179134 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-registration-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179175 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9nnm\" (UniqueName: \"kubernetes.io/projected/95d4dab3-612d-4791-a596-e585151c9990-kube-api-access-g9nnm\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179191 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179197 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-kubernetes\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179230 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-var-lib-kubelet\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.180904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179259 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-run-ovn\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179283 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11a0b4e9-c404-4cf5-b70d-8736829d5821-env-overrides\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179309 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179333 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-sys-fs\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179341 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-run-ovn\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179354 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-run\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179386 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-os-release\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179411 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-cni-dir\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179426 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11a0b4e9-c404-4cf5-b70d-8736829d5821-ovnkube-script-lib\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179435 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-run-multus-certs\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179469 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-systemd-units\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179473 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-run-multus-certs\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179495 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-cni-bin\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179523 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11a0b4e9-c404-4cf5-b70d-8736829d5821-ovnkube-config\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179575 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7560ae93-bfc6-4b22-acde-054add93fe8d-tmp-dir\") pod \"node-resolver-hcdmt\" (UID: \"7560ae93-bfc6-4b22-acde-054add93fe8d\") " pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179597 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-multus-cni-dir\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179599 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.181617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179652 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-systemd-units\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.182380 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179690 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11a0b4e9-c404-4cf5-b70d-8736829d5821-host-cni-bin\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.182380 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179705 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11a0b4e9-c404-4cf5-b70d-8736829d5821-env-overrides\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.182380 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179293 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-host-var-lib-kubelet\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.182380 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.179985 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11a0b4e9-c404-4cf5-b70d-8736829d5821-ovnkube-config\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.182810 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.182794 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11a0b4e9-c404-4cf5-b70d-8736829d5821-ovn-node-metrics-cert\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.186898 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.186880 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862sx\" (UniqueName: \"kubernetes.io/projected/cc54c34b-e67c-4e6a-bd82-a00e09e2d94e-kube-api-access-862sx\") pod \"iptables-alerter-jmq4v\" (UID: \"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e\") " pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.186986 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.186927 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwlht\" (UniqueName: \"kubernetes.io/projected/474dc2d4-8565-47f3-8ebc-dfdf419e11b6-kube-api-access-lwlht\") pod \"multus-7x7hr\" (UID: \"474dc2d4-8565-47f3-8ebc-dfdf419e11b6\") " pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.187391 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.187373 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qw78\" (UniqueName: \"kubernetes.io/projected/11a0b4e9-c404-4cf5-b70d-8736829d5821-kube-api-access-6qw78\") pod \"ovnkube-node-59qz5\" (UID: \"11a0b4e9-c404-4cf5-b70d-8736829d5821\") " pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.198969 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.198929 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c93248cc52546e186356ec011b8aa9.slice/crio-fd2b782f43eb1c800dbdfa497d11fad72c5461c942c4c3d07e421b336d8ed818 WatchSource:0}: Error finding container fd2b782f43eb1c800dbdfa497d11fad72c5461c942c4c3d07e421b336d8ed818: Status 404 returned error can't find the container with id fd2b782f43eb1c800dbdfa497d11fad72c5461c942c4c3d07e421b336d8ed818 Jul 02 18:30:44.199240 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.199222 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e94d30503c9daf0c2ddc17f9303308.slice/crio-b6c12fea637d88dd3924b6c03398aea7869fcff96e33645192897fc58515ed26 WatchSource:0}: Error finding container b6c12fea637d88dd3924b6c03398aea7869fcff96e33645192897fc58515ed26: Status 404 returned error can't find the container with id b6c12fea637d88dd3924b6c03398aea7869fcff96e33645192897fc58515ed26 Jul 02 18:30:44.204269 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.204251 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 18:30:44.232922 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.232881 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" event={"ID":"88c93248cc52546e186356ec011b8aa9","Type":"ContainerStarted","Data":"fd2b782f43eb1c800dbdfa497d11fad72c5461c942c4c3d07e421b336d8ed818"} Jul 02 18:30:44.233782 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.233765 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal" event={"ID":"26e94d30503c9daf0c2ddc17f9303308","Type":"ContainerStarted","Data":"b6c12fea637d88dd3924b6c03398aea7869fcff96e33645192897fc58515ed26"} Jul 02 18:30:44.280085 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280060 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d68dc90b-8983-41f7-a1d7-6c23581adfbe-konnectivity-ca\") pod \"konnectivity-agent-qkjmb\" (UID: \"d68dc90b-8983-41f7-a1d7-6c23581adfbe\") " pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:30:44.280195 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280087 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7560ae93-bfc6-4b22-acde-054add93fe8d-hosts-file\") pod \"node-resolver-hcdmt\" (UID: \"7560ae93-bfc6-4b22-acde-054add93fe8d\") " pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.280195 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280103 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-etc-selinux\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.280195 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280118 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-sysconfig\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.280195 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280157 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-systemd\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280192 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-lib-modules\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280213 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-etc-selinux\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280220 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j68lk\" (UniqueName: \"kubernetes.io/projected/3016b638-7d5e-49c8-9a84-034ff4ab17f7-kube-api-access-j68lk\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280209 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-sysconfig\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280249 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-modprobe-d\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280271 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-sysctl-conf\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280290 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-systemd\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280307 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-system-cni-dir\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280317 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-lib-modules\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280332 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280349 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7560ae93-bfc6-4b22-acde-054add93fe8d-hosts-file\") pod \"node-resolver-hcdmt\" (UID: \"7560ae93-bfc6-4b22-acde-054add93fe8d\") " pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280355 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-system-cni-dir\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.280389 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280357 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlv6\" (UniqueName: \"kubernetes.io/projected/d1038ec5-e712-43d1-bf5e-8370bcefd309-kube-api-access-8tlv6\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280401 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-modprobe-d\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280405 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d103adac-ed3a-485a-90c7-9713adec7aba-kubelet-config\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280445 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-registration-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280447 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-sysctl-conf\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280459 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d103adac-ed3a-485a-90c7-9713adec7aba-kubelet-config\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280471 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9nnm\" (UniqueName: \"kubernetes.io/projected/95d4dab3-612d-4791-a596-e585151c9990-kube-api-access-g9nnm\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280505 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280510 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-kubernetes\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280533 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-registration-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280558 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-kubernetes\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280559 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280600 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-sys-fs\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.280623 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280625 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-run\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280669 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-os-release\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280698 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-sys-fs\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280700 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7560ae93-bfc6-4b22-acde-054add93fe8d-tmp-dir\") pod \"node-resolver-hcdmt\" (UID: \"7560ae93-bfc6-4b22-acde-054add93fe8d\") " pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.281024 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280711 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d68dc90b-8983-41f7-a1d7-6c23581adfbe-konnectivity-ca\") pod \"konnectivity-agent-qkjmb\" (UID: \"d68dc90b-8983-41f7-a1d7-6c23581adfbe\") " pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280718 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-run\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.280724 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs podName:d1038ec5-e712-43d1-bf5e-8370bcefd309 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:44.780705577 +0000 UTC m=+2.067055412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs") pod "network-metrics-daemon-qc7bk" (UID: "d1038ec5-e712-43d1-bf5e-8370bcefd309") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280757 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-os-release\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280764 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280786 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-sys\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280792 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280804 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-host\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280820 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-sys\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280822 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-tuned\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280858 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4de0e2a-fa13-4ffe-8835-9545f4f925af-tmp\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280882 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqnt\" (UniqueName: \"kubernetes.io/projected/7560ae93-bfc6-4b22-acde-054add93fe8d-kube-api-access-ltqnt\") pod \"node-resolver-hcdmt\" (UID: \"7560ae93-bfc6-4b22-acde-054add93fe8d\") " pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280882 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-host\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280898 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d103adac-ed3a-485a-90c7-9713adec7aba-dbus\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280924 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280943 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7560ae93-bfc6-4b22-acde-054add93fe8d-tmp-dir\") pod \"node-resolver-hcdmt\" (UID: \"7560ae93-bfc6-4b22-acde-054add93fe8d\") " pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280955 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3016b638-7d5e-49c8-9a84-034ff4ab17f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.281864 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.280983 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281002 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9f69688-0465-473e-bcab-909e7800ee07-host\") pod \"node-ca-gh2rn\" (UID: \"c9f69688-0465-473e-bcab-909e7800ee07\") " pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.281035 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret podName:d103adac-ed3a-485a-90c7-9713adec7aba nodeName:}" failed. No retries permitted until 2026-07-02 18:30:44.781022791 +0000 UTC m=+2.067372612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret") pod "global-pull-secret-syncer-hjqpc" (UID: "d103adac-ed3a-485a-90c7-9713adec7aba") : object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281047 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281049 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9f69688-0465-473e-bcab-909e7800ee07-host\") pod \"node-ca-gh2rn\" (UID: \"c9f69688-0465-473e-bcab-909e7800ee07\") " pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.280983 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d103adac-ed3a-485a-90c7-9713adec7aba-dbus\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281068 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-sysctl-d\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281106 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d68dc90b-8983-41f7-a1d7-6c23581adfbe-agent-certs\") pod \"konnectivity-agent-qkjmb\" (UID: \"d68dc90b-8983-41f7-a1d7-6c23581adfbe\") " pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281154 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-socket-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281172 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-sysctl-d\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281180 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-device-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281219 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-var-lib-kubelet\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281224 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-device-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281245 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngn2\" (UniqueName: \"kubernetes.io/projected/c4de0e2a-fa13-4ffe-8835-9545f4f925af-kube-api-access-cngn2\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281278 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-cnibin\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281292 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4de0e2a-fa13-4ffe-8835-9545f4f925af-var-lib-kubelet\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.282407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281304 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3016b638-7d5e-49c8-9a84-034ff4ab17f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.282878 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281329 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9f69688-0465-473e-bcab-909e7800ee07-serviceca\") pod \"node-ca-gh2rn\" (UID: \"c9f69688-0465-473e-bcab-909e7800ee07\") " pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.282878 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281355 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njn5\" (UniqueName: \"kubernetes.io/projected/c9f69688-0465-473e-bcab-909e7800ee07-kube-api-access-2njn5\") pod \"node-ca-gh2rn\" (UID: \"c9f69688-0465-473e-bcab-909e7800ee07\") " pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.282878 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281361 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3016b638-7d5e-49c8-9a84-034ff4ab17f7-cnibin\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.282878 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281418 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3016b638-7d5e-49c8-9a84-034ff4ab17f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.282878 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281327 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95d4dab3-612d-4791-a596-e585151c9990-socket-dir\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.282878 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281491 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3016b638-7d5e-49c8-9a84-034ff4ab17f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.282878 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281843 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9f69688-0465-473e-bcab-909e7800ee07-serviceca\") pod \"node-ca-gh2rn\" (UID: \"c9f69688-0465-473e-bcab-909e7800ee07\") " pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.282878 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281917 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3016b638-7d5e-49c8-9a84-034ff4ab17f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.282878 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.281972 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3016b638-7d5e-49c8-9a84-034ff4ab17f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.283407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.283384 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4de0e2a-fa13-4ffe-8835-9545f4f925af-tmp\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.283407 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.283393 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d68dc90b-8983-41f7-a1d7-6c23581adfbe-agent-certs\") pod \"konnectivity-agent-qkjmb\" (UID: \"d68dc90b-8983-41f7-a1d7-6c23581adfbe\") " pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:30:44.283513 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.283419 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4de0e2a-fa13-4ffe-8835-9545f4f925af-etc-tuned\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.288039 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.288018 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 18:30:44.288140 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.288043 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 18:30:44.288140 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.288057 2552 projected.go:196] Error preparing data for projected volume kube-api-access-wlhcf for pod openshift-network-diagnostics/network-check-target-n98x8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:44.288140 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.288131 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf podName:91c774ed-1026-4a13-91ba-61cb58ae1bca nodeName:}" failed. No retries permitted until 2026-07-02 18:30:44.788113736 +0000 UTC m=+2.074463588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wlhcf" (UniqueName: "kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf") pod "network-check-target-n98x8" (UID: "91c774ed-1026-4a13-91ba-61cb58ae1bca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:44.288958 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.288936 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9nnm\" (UniqueName: \"kubernetes.io/projected/95d4dab3-612d-4791-a596-e585151c9990-kube-api-access-g9nnm\") pod \"aws-ebs-csi-driver-node-9776k\" (UID: \"95d4dab3-612d-4791-a596-e585151c9990\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.289134 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.289113 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqnt\" (UniqueName: \"kubernetes.io/projected/7560ae93-bfc6-4b22-acde-054add93fe8d-kube-api-access-ltqnt\") pod \"node-resolver-hcdmt\" (UID: \"7560ae93-bfc6-4b22-acde-054add93fe8d\") " pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.289299 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.289280 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tlv6\" (UniqueName: \"kubernetes.io/projected/d1038ec5-e712-43d1-bf5e-8370bcefd309-kube-api-access-8tlv6\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:44.289340 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.289313 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j68lk\" (UniqueName: \"kubernetes.io/projected/3016b638-7d5e-49c8-9a84-034ff4ab17f7-kube-api-access-j68lk\") pod \"multus-additional-cni-plugins-r52rg\" (UID: \"3016b638-7d5e-49c8-9a84-034ff4ab17f7\") " pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.290172 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.290157 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngn2\" (UniqueName: \"kubernetes.io/projected/c4de0e2a-fa13-4ffe-8835-9545f4f925af-kube-api-access-cngn2\") pod \"tuned-p4w84\" (UID: \"c4de0e2a-fa13-4ffe-8835-9545f4f925af\") " pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.290251 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.290237 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njn5\" (UniqueName: \"kubernetes.io/projected/c9f69688-0465-473e-bcab-909e7800ee07-kube-api-access-2njn5\") pod \"node-ca-gh2rn\" (UID: \"c9f69688-0465-473e-bcab-909e7800ee07\") " pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.404537 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.404469 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7x7hr" Jul 02 18:30:44.404928 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.404911 2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 18:30:44.420112 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.420096 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jmq4v" Jul 02 18:30:44.426528 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.426507 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc54c34b_e67c_4e6a_bd82_a00e09e2d94e.slice/crio-f13b6074010cd431e07d4f80d6e3fa2772ae0a6e7e74c57ae271e4919c3f3c08 WatchSource:0}: Error finding container f13b6074010cd431e07d4f80d6e3fa2772ae0a6e7e74c57ae271e4919c3f3c08: Status 404 returned error can't find the container with id f13b6074010cd431e07d4f80d6e3fa2772ae0a6e7e74c57ae271e4919c3f3c08 Jul 02 18:30:44.438882 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.438862 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:30:44.445681 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.445660 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a0b4e9_c404_4cf5_b70d_8736829d5821.slice/crio-254e94757b06327c247283283052915bb7e12344e163a6c70dc160e0c4be8af3 WatchSource:0}: Error finding container 254e94757b06327c247283283052915bb7e12344e163a6c70dc160e0c4be8af3: Status 404 returned error can't find the container with id 254e94757b06327c247283283052915bb7e12344e163a6c70dc160e0c4be8af3 Jul 02 18:30:44.450783 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.450764 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:30:44.458364 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.458345 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68dc90b_8983_41f7_a1d7_6c23581adfbe.slice/crio-d943bfad5b52b9fccf5f0388a093d6dece4ec7099f66bea2a290d973f9ecaacf WatchSource:0}: Error finding container d943bfad5b52b9fccf5f0388a093d6dece4ec7099f66bea2a290d973f9ecaacf: Status 404 returned error can't find the container with id d943bfad5b52b9fccf5f0388a093d6dece4ec7099f66bea2a290d973f9ecaacf Jul 02 18:30:44.465349 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.465335 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" Jul 02 18:30:44.472564 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.472543 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d4dab3_612d_4791_a596_e585151c9990.slice/crio-b74d52215764c168cefa315a5cf50ea982304094eb2045aa685f66c4214842f8 WatchSource:0}: Error finding container b74d52215764c168cefa315a5cf50ea982304094eb2045aa685f66c4214842f8: Status 404 returned error can't find the container with id b74d52215764c168cefa315a5cf50ea982304094eb2045aa685f66c4214842f8 Jul 02 18:30:44.478023 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.478007 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p4w84" Jul 02 18:30:44.484821 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.484804 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4de0e2a_fa13_4ffe_8835_9545f4f925af.slice/crio-d30d7577cdcde3af3aa8430ff677bcc5f19452007e08b970d010897a161f7b8c WatchSource:0}: Error finding container d30d7577cdcde3af3aa8430ff677bcc5f19452007e08b970d010897a161f7b8c: Status 404 returned error can't find the container with id d30d7577cdcde3af3aa8430ff677bcc5f19452007e08b970d010897a161f7b8c Jul 02 18:30:44.510504 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.510485 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r52rg" Jul 02 18:30:44.517179 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.517151 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hcdmt" Jul 02 18:30:44.519132 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.519114 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3016b638_7d5e_49c8_9a84_034ff4ab17f7.slice/crio-66b0da2111e82d276a856ba8aa62a8f84b69371a2a9361844edf5bbe0a160a87 WatchSource:0}: Error finding container 66b0da2111e82d276a856ba8aa62a8f84b69371a2a9361844edf5bbe0a160a87: Status 404 returned error can't find the container with id 66b0da2111e82d276a856ba8aa62a8f84b69371a2a9361844edf5bbe0a160a87 Jul 02 18:30:44.520911 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.520895 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gh2rn" Jul 02 18:30:44.524422 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.524402 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7560ae93_bfc6_4b22_acde_054add93fe8d.slice/crio-a4b038b8a4c972e4a223980f99776c7dd2db964463555fcac287cdcf4621ad04 WatchSource:0}: Error finding container a4b038b8a4c972e4a223980f99776c7dd2db964463555fcac287cdcf4621ad04: Status 404 returned error can't find the container with id a4b038b8a4c972e4a223980f99776c7dd2db964463555fcac287cdcf4621ad04 Jul 02 18:30:44.531845 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:30:44.531823 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f69688_0465_473e_bcab_909e7800ee07.slice/crio-8fd1b390866c90c73c6afdf88b6179e18829602600872daf8548887078abb418 WatchSource:0}: Error finding container 8fd1b390866c90c73c6afdf88b6179e18829602600872daf8548887078abb418: Status 404 returned error can't find the container with id 8fd1b390866c90c73c6afdf88b6179e18829602600872daf8548887078abb418 Jul 02 18:30:44.785588 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.785505 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:44.785588 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.785568 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:44.785881 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.785749 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:44.785881 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.785809 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret podName:d103adac-ed3a-485a-90c7-9713adec7aba nodeName:}" failed. No retries permitted until 2026-07-02 18:30:45.785790127 +0000 UTC m=+3.072139951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret") pod "global-pull-secret-syncer-hjqpc" (UID: "d103adac-ed3a-485a-90c7-9713adec7aba") : object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:44.786242 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.786224 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:44.786309 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.786280 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs podName:d1038ec5-e712-43d1-bf5e-8370bcefd309 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:45.786264119 +0000 UTC m=+3.072613944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs") pod "network-metrics-daemon-qc7bk" (UID: "d1038ec5-e712-43d1-bf5e-8370bcefd309") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:44.886417 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:44.886382 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:44.886601 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.886537 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 18:30:44.886601 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.886556 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 18:30:44.886601 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.886568 2552 projected.go:196] Error preparing data for projected volume kube-api-access-wlhcf for pod openshift-network-diagnostics/network-check-target-n98x8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:44.886798 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:44.886626 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf podName:91c774ed-1026-4a13-91ba-61cb58ae1bca nodeName:}" failed. No retries permitted until 2026-07-02 18:30:45.886607629 +0000 UTC m=+3.172957475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlhcf" (UniqueName: "kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf") pod "network-check-target-n98x8" (UID: "91c774ed-1026-4a13-91ba-61cb58ae1bca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:45.113726 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.113548 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 18:25:44 +0000 UTC" deadline="2027-12-29 15:59:18.250928951 +0000 UTC" Jul 02 18:30:45.113726 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.113613 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13077h28m33.137320353s" Jul 02 18:30:45.141251 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.141223 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 18:30:45.263231 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.263193 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gh2rn" event={"ID":"c9f69688-0465-473e-bcab-909e7800ee07","Type":"ContainerStarted","Data":"8fd1b390866c90c73c6afdf88b6179e18829602600872daf8548887078abb418"} Jul 02 18:30:45.269592 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.269564 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r52rg" event={"ID":"3016b638-7d5e-49c8-9a84-034ff4ab17f7","Type":"ContainerStarted","Data":"66b0da2111e82d276a856ba8aa62a8f84b69371a2a9361844edf5bbe0a160a87"} Jul 02 18:30:45.272266 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.272236 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p4w84" event={"ID":"c4de0e2a-fa13-4ffe-8835-9545f4f925af","Type":"ContainerStarted","Data":"d30d7577cdcde3af3aa8430ff677bcc5f19452007e08b970d010897a161f7b8c"} Jul 02 18:30:45.298854 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.298721 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" event={"ID":"95d4dab3-612d-4791-a596-e585151c9990","Type":"ContainerStarted","Data":"b74d52215764c168cefa315a5cf50ea982304094eb2045aa685f66c4214842f8"} Jul 02 18:30:45.302916 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.302883 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qkjmb" event={"ID":"d68dc90b-8983-41f7-a1d7-6c23581adfbe","Type":"ContainerStarted","Data":"d943bfad5b52b9fccf5f0388a093d6dece4ec7099f66bea2a290d973f9ecaacf"} Jul 02 18:30:45.305431 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.305338 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" event={"ID":"11a0b4e9-c404-4cf5-b70d-8736829d5821","Type":"ContainerStarted","Data":"254e94757b06327c247283283052915bb7e12344e163a6c70dc160e0c4be8af3"} Jul 02 18:30:45.313105 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.313068 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jmq4v" event={"ID":"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e","Type":"ContainerStarted","Data":"f13b6074010cd431e07d4f80d6e3fa2772ae0a6e7e74c57ae271e4919c3f3c08"} Jul 02 18:30:45.324760 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.324730 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hcdmt" event={"ID":"7560ae93-bfc6-4b22-acde-054add93fe8d","Type":"ContainerStarted","Data":"a4b038b8a4c972e4a223980f99776c7dd2db964463555fcac287cdcf4621ad04"} Jul 02 18:30:45.339216 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.339066 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7x7hr" event={"ID":"474dc2d4-8565-47f3-8ebc-dfdf419e11b6","Type":"ContainerStarted","Data":"411802c0eab78e75048b814ad00b77f3dc7c353d1a093e71ab70f7e66b757ab7"} Jul 02 18:30:45.795531 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.795497 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:45.795750 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.795560 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:45.795750 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:45.795721 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:45.795876 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:45.795784 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret podName:d103adac-ed3a-485a-90c7-9713adec7aba nodeName:}" failed. No retries permitted until 2026-07-02 18:30:47.795764151 +0000 UTC m=+5.082113977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret") pod "global-pull-secret-syncer-hjqpc" (UID: "d103adac-ed3a-485a-90c7-9713adec7aba") : object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:45.796182 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:45.796162 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:45.796249 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:45.796223 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs podName:d1038ec5-e712-43d1-bf5e-8370bcefd309 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:47.796207358 +0000 UTC m=+5.082557183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs") pod "network-metrics-daemon-qc7bk" (UID: "d1038ec5-e712-43d1-bf5e-8370bcefd309") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:45.896167 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:45.896135 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:45.896358 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:45.896342 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 18:30:45.896425 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:45.896369 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 18:30:45.896425 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:45.896382 2552 projected.go:196] Error preparing data for projected volume kube-api-access-wlhcf for pod openshift-network-diagnostics/network-check-target-n98x8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:45.896521 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:45.896445 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf podName:91c774ed-1026-4a13-91ba-61cb58ae1bca nodeName:}" failed. No retries permitted until 2026-07-02 18:30:47.896427122 +0000 UTC m=+5.182776949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlhcf" (UniqueName: "kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf") pod "network-check-target-n98x8" (UID: "91c774ed-1026-4a13-91ba-61cb58ae1bca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:46.114168 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:46.114081 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 18:25:44 +0000 UTC" deadline="2028-03-17 06:07:14.535016348 +0000 UTC" Jul 02 18:30:46.114168 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:46.114117 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14963h36m28.420903823s" Jul 02 18:30:46.132239 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:46.132211 2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 18:30:46.231302 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:46.231275 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:46.231481 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:46.231399 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:30:46.231987 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:46.231962 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:46.232130 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:46.232109 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:30:46.232191 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:46.232184 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:46.232273 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:46.232254 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:30:47.813978 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:47.813944 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:47.814444 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:47.814016 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:47.814444 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:47.814171 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:47.814444 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:47.814315 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret podName:d103adac-ed3a-485a-90c7-9713adec7aba nodeName:}" failed. No retries permitted until 2026-07-02 18:30:51.814285606 +0000 UTC m=+9.100635430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret") pod "global-pull-secret-syncer-hjqpc" (UID: "d103adac-ed3a-485a-90c7-9713adec7aba") : object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:47.814629 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:47.814610 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:47.814697 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:47.814689 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs podName:d1038ec5-e712-43d1-bf5e-8370bcefd309 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:51.814674913 +0000 UTC m=+9.101024734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs") pod "network-metrics-daemon-qc7bk" (UID: "d1038ec5-e712-43d1-bf5e-8370bcefd309") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:47.915360 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:47.914810 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:47.915360 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:47.914996 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 18:30:47.915360 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:47.915017 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 18:30:47.915360 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:47.915028 2552 projected.go:196] Error preparing data for projected volume kube-api-access-wlhcf for pod openshift-network-diagnostics/network-check-target-n98x8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:47.915360 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:47.915081 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf podName:91c774ed-1026-4a13-91ba-61cb58ae1bca nodeName:}" failed. No retries permitted until 2026-07-02 18:30:51.915063477 +0000 UTC m=+9.201413335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlhcf" (UniqueName: "kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf") pod "network-check-target-n98x8" (UID: "91c774ed-1026-4a13-91ba-61cb58ae1bca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:48.230340 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:48.230261 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:48.230506 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:48.230388 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:30:48.230572 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:48.230544 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:48.230572 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:48.230556 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:48.230781 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:48.230729 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:30:48.230845 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:48.230825 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:30:50.230687 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:50.230619 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:50.231167 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:50.230747 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:50.231167 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:50.230764 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:30:50.231167 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:50.230876 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:30:50.231167 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:50.230940 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:50.231167 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:50.231013 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:30:51.855149 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:51.855110 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:51.855581 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:51.855206 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:51.855581 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:51.855284 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:51.855581 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:51.855333 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:51.855581 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:51.855370 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs podName:d1038ec5-e712-43d1-bf5e-8370bcefd309 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:59.85534843 +0000 UTC m=+17.141698250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs") pod "network-metrics-daemon-qc7bk" (UID: "d1038ec5-e712-43d1-bf5e-8370bcefd309") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:51.855581 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:51.855393 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret podName:d103adac-ed3a-485a-90c7-9713adec7aba nodeName:}" failed. No retries permitted until 2026-07-02 18:30:59.855382046 +0000 UTC m=+17.141731869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret") pod "global-pull-secret-syncer-hjqpc" (UID: "d103adac-ed3a-485a-90c7-9713adec7aba") : object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:51.955842 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:51.955805 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:51.956026 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:51.956009 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 18:30:51.956109 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:51.956033 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 18:30:51.956109 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:51.956046 2552 projected.go:196] Error preparing data for projected volume kube-api-access-wlhcf for pod openshift-network-diagnostics/network-check-target-n98x8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:51.956209 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:51.956109 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf podName:91c774ed-1026-4a13-91ba-61cb58ae1bca nodeName:}" failed. No retries permitted until 2026-07-02 18:30:59.956089504 +0000 UTC m=+17.242439326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlhcf" (UniqueName: "kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf") pod "network-check-target-n98x8" (UID: "91c774ed-1026-4a13-91ba-61cb58ae1bca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:30:52.231349 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:52.231263 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:52.231494 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:52.231411 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:30:52.231565 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:52.231499 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:52.231614 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:52.231583 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:30:52.231686 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:52.231630 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:52.231763 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:52.231733 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:30:54.230623 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:54.230585 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:54.230623 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:54.230613 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:54.231136 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:54.230714 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:54.231136 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:54.230731 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:30:54.231136 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:54.230787 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:30:54.231136 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:54.230880 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:30:56.231350 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:56.231260 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:56.231783 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:56.231391 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:30:56.231783 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:56.231409 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:56.231783 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:56.231400 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:56.231783 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:56.231520 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:30:56.231783 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:56.231601 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:30:58.231433 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:58.231395 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:58.231932 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:58.231539 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:58.231932 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:58.231553 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:30:58.231932 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:58.231585 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:30:58.231932 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:58.231686 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:30:58.231932 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:58.231763 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:30:59.908401 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:59.908323 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:30:59.908896 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:30:59.908405 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:30:59.908896 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:59.908508 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:59.908896 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:59.908526 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:30:59.908896 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:59.908573 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret podName:d103adac-ed3a-485a-90c7-9713adec7aba nodeName:}" failed. No retries permitted until 2026-07-02 18:31:15.908557058 +0000 UTC m=+33.194906884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret") pod "global-pull-secret-syncer-hjqpc" (UID: "d103adac-ed3a-485a-90c7-9713adec7aba") : object "kube-system"/"original-pull-secret" not registered Jul 02 18:30:59.908896 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:30:59.908593 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs podName:d1038ec5-e712-43d1-bf5e-8370bcefd309 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:15.908583367 +0000 UTC m=+33.194933194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs") pod "network-metrics-daemon-qc7bk" (UID: "d1038ec5-e712-43d1-bf5e-8370bcefd309") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:31:00.009259 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:00.009220 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:00.009440 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:00.009384 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 18:31:00.009440 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:00.009405 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 18:31:00.009440 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:00.009416 2552 projected.go:196] Error preparing data for projected volume kube-api-access-wlhcf for pod openshift-network-diagnostics/network-check-target-n98x8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:31:00.009592 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:00.009483 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf podName:91c774ed-1026-4a13-91ba-61cb58ae1bca nodeName:}" failed. No retries permitted until 2026-07-02 18:31:16.009464035 +0000 UTC m=+33.295813881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlhcf" (UniqueName: "kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf") pod "network-check-target-n98x8" (UID: "91c774ed-1026-4a13-91ba-61cb58ae1bca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:31:00.230553 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:00.230525 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:00.230766 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:00.230525 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:00.230766 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:00.230648 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:00.230766 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:00.230525 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:00.230899 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:00.230720 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:00.230899 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:00.230832 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:02.231325 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:02.231285 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:02.231325 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:02.231311 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:02.231899 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:02.231410 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:02.231899 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:02.231507 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:02.231899 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:02.231552 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:02.231899 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:02.231624 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:04.230828 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:04.230799 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:04.231143 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:04.230845 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:04.231143 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:04.230845 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:04.231143 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:04.230903 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:04.231143 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:04.231001 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:04.231143 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:04.231078 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:04.396121 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:04.396093 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal" event={"ID":"26e94d30503c9daf0c2ddc17f9303308","Type":"ContainerStarted","Data":"d9532ceb61eb8efaf09cee25987e409aee801f4b9bc1412da30c8e45e712579a"} Jul 02 18:31:04.401344 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:04.401311 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p4w84" event={"ID":"c4de0e2a-fa13-4ffe-8835-9545f4f925af","Type":"ContainerStarted","Data":"d9c4af9e5779962549904fcd845cc0cada6a1d039478e8435a04215158335532"} Jul 02 18:31:04.411970 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:04.411894 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" event={"ID":"11a0b4e9-c404-4cf5-b70d-8736829d5821","Type":"ContainerStarted","Data":"2d625b3a18c67dbb47c84f32966dbf46b52c160b6bdcc5b91a1924b6a42a38ee"} Jul 02 18:31:04.460090 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:04.459816 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p4w84" podStartSLOduration=1.865466788 podStartE2EDuration="21.459793867s" totalImagesPullingTime="19.594327079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:04.459496447 +0000 UTC m=+21.745846291" watchObservedRunningTime="2026-07-02 18:31:04.459793867 +0000 UTC m=+21.746143712" Jul 02 18:31:04.460494 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:04.460450 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-23.ec2.internal" podStartSLOduration=21.460435111 podStartE2EDuration="21.460435111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:04.42415829 +0000 UTC m=+21.710508133" watchObservedRunningTime="2026-07-02 18:31:04.460435111 +0000 UTC m=+21.746784955" Jul 02 18:31:05.416377 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.416173 2552 generic.go:356] "Generic (PLEG): container finished" podID="88c93248cc52546e186356ec011b8aa9" containerID="d2ab03ff7c8485a070bc7454b885d7f56cfbbbd0ed7952862b59d9f4b00481bc" exitCode=0 Jul 02 18:31:05.417287 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.416272 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" event={"ID":"88c93248cc52546e186356ec011b8aa9","Type":"ContainerDied","Data":"d2ab03ff7c8485a070bc7454b885d7f56cfbbbd0ed7952862b59d9f4b00481bc"} Jul 02 18:31:05.418068 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.418041 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hcdmt" event={"ID":"7560ae93-bfc6-4b22-acde-054add93fe8d","Type":"ContainerStarted","Data":"4732016aa050f8e50a4f492d064a2b7673b12b8644aad4ca1faa80f2f73b4c84"} Jul 02 18:31:05.419711 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.419659 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7x7hr" event={"ID":"474dc2d4-8565-47f3-8ebc-dfdf419e11b6","Type":"ContainerStarted","Data":"bf20e2054aebf06a0c0691b9401f7f9651e7880afd7b5c5dc296a309db7cc837"} Jul 02 18:31:05.421214 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.421187 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gh2rn" event={"ID":"c9f69688-0465-473e-bcab-909e7800ee07","Type":"ContainerStarted","Data":"b9f69cc6819472d887f84a9a65141eded6379542d085a37e877ba54c17e2f637"} Jul 02 18:31:05.422896 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.422873 2552 generic.go:356] "Generic (PLEG): container finished" podID="3016b638-7d5e-49c8-9a84-034ff4ab17f7" containerID="dfdc57c528965edc17bc83f30171aca50f8af122ca102b7d288ef4f576da303e" exitCode=0 Jul 02 18:31:05.422974 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.422932 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r52rg" event={"ID":"3016b638-7d5e-49c8-9a84-034ff4ab17f7","Type":"ContainerDied","Data":"dfdc57c528965edc17bc83f30171aca50f8af122ca102b7d288ef4f576da303e"} Jul 02 18:31:05.424752 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.424731 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" event={"ID":"95d4dab3-612d-4791-a596-e585151c9990","Type":"ContainerStarted","Data":"4b899845520e2b743fa7da4ea76a74db7e01026b1eedcd4313d05f4d6b4b4e71"} Jul 02 18:31:05.426241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.426205 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qkjmb" event={"ID":"d68dc90b-8983-41f7-a1d7-6c23581adfbe","Type":"ContainerStarted","Data":"37d4bbb19261237271403a8069e8c3ca8d0195bc9c1c6c93a3d6b45b08adc465"} Jul 02 18:31:05.429671 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.429554 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" event={"ID":"11a0b4e9-c404-4cf5-b70d-8736829d5821","Type":"ContainerStarted","Data":"acdb4caf56145443fcb926d6247cdbeb4a223b37cd30348d5e573441ae671713"} Jul 02 18:31:05.429671 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.429627 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" event={"ID":"11a0b4e9-c404-4cf5-b70d-8736829d5821","Type":"ContainerStarted","Data":"8b3fe019ee2b46ad2b19ab0b6b056c47124aeda8c0e1c8b86064188d45508ad7"} Jul 02 18:31:05.429814 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.429679 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" event={"ID":"11a0b4e9-c404-4cf5-b70d-8736829d5821","Type":"ContainerStarted","Data":"66c61d90d661563d1e5eb27ef0db8d3bc514b2ab010ccb73e54e9594a8341db7"} Jul 02 18:31:05.429814 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.429723 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" event={"ID":"11a0b4e9-c404-4cf5-b70d-8736829d5821","Type":"ContainerStarted","Data":"24c43586e6a9966e75e79156035fab91a3df9bc0bdb944fd49113f17f2d54f39"} Jul 02 18:31:05.429814 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.429756 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" event={"ID":"11a0b4e9-c404-4cf5-b70d-8736829d5821","Type":"ContainerStarted","Data":"8bd10e214a5405e8a7e0c3ffe7dd8fef073b0bb168cde91bf98a2f959da64d42"} Jul 02 18:31:05.465401 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.465265 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7x7hr" podStartSLOduration=2.586150311 podStartE2EDuration="22.465245577s" totalImagesPullingTime="19.879095266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:05.464478501 +0000 UTC m=+22.750828345" watchObservedRunningTime="2026-07-02 18:31:05.465245577 +0000 UTC m=+22.751595419" Jul 02 18:31:05.477762 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.477719 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qkjmb" podStartSLOduration=2.7789527229999997 podStartE2EDuration="22.47770251s" totalImagesPullingTime="19.698749787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:05.477623893 +0000 UTC m=+22.763973735" watchObservedRunningTime="2026-07-02 18:31:05.47770251 +0000 UTC m=+22.764052351" Jul 02 18:31:05.492701 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.492657 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hcdmt" podStartSLOduration=2.854222336 podStartE2EDuration="22.492626226s" totalImagesPullingTime="19.63840389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:05.492058693 +0000 UTC m=+22.778408537" watchObservedRunningTime="2026-07-02 18:31:05.492626226 +0000 UTC m=+22.778976066" Jul 02 18:31:05.508997 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.508944 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gh2rn" podStartSLOduration=2.883626162 podStartE2EDuration="22.508930621s" totalImagesPullingTime="19.625304459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:05.508698905 +0000 UTC m=+22.795048748" watchObservedRunningTime="2026-07-02 18:31:05.508930621 +0000 UTC m=+22.795280463" Jul 02 18:31:05.956037 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:05.956013 2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 18:31:06.153268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.153125 2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T18:31:05.956031278Z","UUID":"7579468e-799a-494a-9bd0-fba306ebe35b","Handler":null,"Name":"","Endpoint":""} Jul 02 18:31:06.154742 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.154719 2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 18:31:06.154880 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.154765 2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 18:31:06.231035 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.231004 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:06.231251 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.231111 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:06.231251 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:06.231121 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:06.231251 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:06.231226 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:06.231459 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.231279 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:06.231459 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:06.231376 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:06.433295 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.433259 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jmq4v" event={"ID":"cc54c34b-e67c-4e6a-bd82-a00e09e2d94e","Type":"ContainerStarted","Data":"de00f72b799b71fdcfa69c18f387c4e6cd826b0117543a221798ca8f24a18ac0"} Jul 02 18:31:06.435272 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.435240 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" event={"ID":"88c93248cc52546e186356ec011b8aa9","Type":"ContainerStarted","Data":"2833177e98a1139891fd186092e712ec853d5d40dbf145965d488af6544d677f"} Jul 02 18:31:06.437278 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.437250 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" event={"ID":"95d4dab3-612d-4791-a596-e585151c9990","Type":"ContainerStarted","Data":"e22600acef88ed8835ba01360812b2eb328cb2afd4a15ead5d1f5fe7d5b0fda5"} Jul 02 18:31:06.448927 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:06.448863 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jmq4v" podStartSLOduration=3.744023174 podStartE2EDuration="23.448843828s" totalImagesPullingTime="19.704820654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:06.447954577 +0000 UTC m=+23.734304420" watchObservedRunningTime="2026-07-02 18:31:06.448843828 +0000 UTC m=+23.735193671" Jul 02 18:31:07.443995 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:07.443915 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" event={"ID":"11a0b4e9-c404-4cf5-b70d-8736829d5821","Type":"ContainerStarted","Data":"4658af8ad25c4db4799f6c755075b1229dad509316d79715d7551253b396a6be"} Jul 02 18:31:08.231182 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.231142 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:08.231437 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.231208 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:08.231437 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:08.231297 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:08.231437 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.231308 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:08.231575 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:08.231433 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:08.231575 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:08.231507 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:08.354353 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.354324 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:31:08.355018 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.354985 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:31:08.368135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.368095 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-23.ec2.internal" podStartSLOduration=25.368083135 podStartE2EDuration="25.368083135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:06.4611277 +0000 UTC m=+23.747477558" watchObservedRunningTime="2026-07-02 18:31:08.368083135 +0000 UTC m=+25.654432976" Jul 02 18:31:08.449033 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.448995 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" event={"ID":"95d4dab3-612d-4791-a596-e585151c9990","Type":"ContainerStarted","Data":"6934854c14320e875cd782efc6474b736758b499d6f1930de4857fb214c4eaf7"} Jul 02 18:31:08.449507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.449400 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:31:08.449825 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.449804 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qkjmb" Jul 02 18:31:08.466050 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:08.466007 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9776k" podStartSLOduration=3.1920808210000002 podStartE2EDuration="25.465993407s" totalImagesPullingTime="22.273912586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:08.465511643 +0000 UTC m=+25.751861486" watchObservedRunningTime="2026-07-02 18:31:08.465993407 +0000 UTC m=+25.752343287" Jul 02 18:31:10.231049 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.230856 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:10.231744 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.230893 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:10.231744 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:10.231139 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:10.231744 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.230904 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:10.231744 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:10.231197 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:10.231744 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:10.231285 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:10.454726 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.454693 2552 generic.go:356] "Generic (PLEG): container finished" podID="3016b638-7d5e-49c8-9a84-034ff4ab17f7" containerID="6f91bb73f5259dc9bf1c7532809b09cb13cb14d803d4840eba102861c6eb94b7" exitCode=0 Jul 02 18:31:10.454909 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.454788 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r52rg" event={"ID":"3016b638-7d5e-49c8-9a84-034ff4ab17f7","Type":"ContainerDied","Data":"6f91bb73f5259dc9bf1c7532809b09cb13cb14d803d4840eba102861c6eb94b7"} Jul 02 18:31:10.462013 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.461981 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" event={"ID":"11a0b4e9-c404-4cf5-b70d-8736829d5821","Type":"ContainerStarted","Data":"a0415c5ac36b0d0c6f039c79a4a4b67092534d0a281815956c66e3bbb14a52c7"} Jul 02 18:31:10.462292 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.462276 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:31:10.462363 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.462330 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:31:10.476803 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.476784 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:31:10.502025 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:10.501946 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" podStartSLOduration=7.773281308 podStartE2EDuration="27.50193227s" totalImagesPullingTime="19.728650962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:10.501928241 +0000 UTC m=+27.788278083" watchObservedRunningTime="2026-07-02 18:31:10.50193227 +0000 UTC m=+27.788282111" Jul 02 18:31:11.464685 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:11.464656 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:31:11.479517 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:11.479496 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:31:12.231050 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:12.231020 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:12.231195 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:12.231106 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:12.231195 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:12.231027 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:12.231195 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:12.231160 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:12.231195 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:12.231027 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:12.231363 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:12.231220 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:12.467736 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:12.467685 2552 generic.go:356] "Generic (PLEG): container finished" podID="3016b638-7d5e-49c8-9a84-034ff4ab17f7" containerID="f2f3f365021426166f4f1de4324844dd94b2d3189cb480e6681150dd6b4b4ad5" exitCode=0 Jul 02 18:31:12.468081 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:12.467776 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r52rg" event={"ID":"3016b638-7d5e-49c8-9a84-034ff4ab17f7","Type":"ContainerDied","Data":"f2f3f365021426166f4f1de4324844dd94b2d3189cb480e6681150dd6b4b4ad5"} Jul 02 18:31:14.230483 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:14.230306 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:14.230858 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:14.230306 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:14.230858 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:14.230555 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:14.230858 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:14.230306 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:14.230858 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:14.230629 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:14.230858 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:14.230710 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:14.474317 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:14.474284 2552 generic.go:356] "Generic (PLEG): container finished" podID="3016b638-7d5e-49c8-9a84-034ff4ab17f7" containerID="7fb975ae0fee2957c38f641677d354c521988e978ba15e04db784541c7d6396d" exitCode=0 Jul 02 18:31:14.474484 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:14.474337 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r52rg" event={"ID":"3016b638-7d5e-49c8-9a84-034ff4ab17f7","Type":"ContainerDied","Data":"7fb975ae0fee2957c38f641677d354c521988e978ba15e04db784541c7d6396d"} Jul 02 18:31:15.929751 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:15.929717 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:15.930176 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:15.929795 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:15.930176 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:15.929868 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 18:31:15.930176 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:15.929891 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:31:15.930176 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:15.929934 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret podName:d103adac-ed3a-485a-90c7-9713adec7aba nodeName:}" failed. No retries permitted until 2026-07-02 18:31:47.929918795 +0000 UTC m=+65.216268616 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret") pod "global-pull-secret-syncer-hjqpc" (UID: "d103adac-ed3a-485a-90c7-9713adec7aba") : object "kube-system"/"original-pull-secret" not registered Jul 02 18:31:15.930176 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:15.929948 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs podName:d1038ec5-e712-43d1-bf5e-8370bcefd309 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:47.929941595 +0000 UTC m=+65.216291415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs") pod "network-metrics-daemon-qc7bk" (UID: "d1038ec5-e712-43d1-bf5e-8370bcefd309") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 18:31:16.030371 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:16.030339 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:16.030532 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:16.030480 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 18:31:16.030532 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:16.030503 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 18:31:16.030532 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:16.030515 2552 projected.go:196] Error preparing data for projected volume kube-api-access-wlhcf for pod openshift-network-diagnostics/network-check-target-n98x8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:31:16.030694 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:16.030575 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf podName:91c774ed-1026-4a13-91ba-61cb58ae1bca nodeName:}" failed. No retries permitted until 2026-07-02 18:31:48.030557361 +0000 UTC m=+65.316907182 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlhcf" (UniqueName: "kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf") pod "network-check-target-n98x8" (UID: "91c774ed-1026-4a13-91ba-61cb58ae1bca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 18:31:16.230730 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:16.230647 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:16.230730 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:16.230673 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:16.230936 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:16.230783 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:16.230995 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:16.230927 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:16.230995 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:16.230987 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:16.231097 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:16.231054 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:18.231199 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:18.231173 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:18.231528 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:18.231267 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:18.231528 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:18.231303 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:18.231528 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:18.231374 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:18.231528 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:18.231267 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:18.231681 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:18.231535 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:18.486771 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:18.486741 2552 generic.go:356] "Generic (PLEG): container finished" podID="3016b638-7d5e-49c8-9a84-034ff4ab17f7" containerID="0a27e977505249acbe969e5f541cd34c52408a2edde7ec7b44b3444435147ac7" exitCode=0 Jul 02 18:31:18.486927 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:18.486798 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r52rg" event={"ID":"3016b638-7d5e-49c8-9a84-034ff4ab17f7","Type":"ContainerDied","Data":"0a27e977505249acbe969e5f541cd34c52408a2edde7ec7b44b3444435147ac7"} Jul 02 18:31:19.492050 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:19.492017 2552 generic.go:356] "Generic (PLEG): container finished" podID="3016b638-7d5e-49c8-9a84-034ff4ab17f7" containerID="8959b7f4d8ee711f5843f00e2c330c2617b6dd3ee75807eb13548c27e3924261" exitCode=0 Jul 02 18:31:19.492398 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:19.492076 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r52rg" event={"ID":"3016b638-7d5e-49c8-9a84-034ff4ab17f7","Type":"ContainerDied","Data":"8959b7f4d8ee711f5843f00e2c330c2617b6dd3ee75807eb13548c27e3924261"} Jul 02 18:31:20.230937 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:20.230908 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:20.230937 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:20.230924 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:20.231116 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:20.230923 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:20.231116 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:20.231008 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:20.231116 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:20.231066 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:20.231204 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:20.231145 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:20.499759 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:20.499661 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r52rg" event={"ID":"3016b638-7d5e-49c8-9a84-034ff4ab17f7","Type":"ContainerStarted","Data":"1c28d668f054345727776e8a637081725e00ebe59e0dd5e91230521373ed8021"} Jul 02 18:31:20.536035 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:20.535990 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r52rg" podStartSLOduration=7.856202362 podStartE2EDuration="37.535977376s" totalImagesPullingTime="29.679775014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:20.534673756 +0000 UTC m=+37.821023595" watchObservedRunningTime="2026-07-02 18:31:20.535977376 +0000 UTC m=+37.822327219" Jul 02 18:31:22.230727 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:22.230692 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:22.231085 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:22.230786 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:22.231085 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:22.230791 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:22.231085 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:22.230868 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:22.231085 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:22.230927 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:22.231085 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:22.230788 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:23.777223 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:23.777057 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n98x8"] Jul 02 18:31:23.777223 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:23.777196 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:23.777753 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:23.777292 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:23.778123 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:23.778097 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hjqpc"] Jul 02 18:31:23.778233 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:23.778188 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:23.778290 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:23.778261 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:23.785597 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:23.785444 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qc7bk"] Jul 02 18:31:23.785701 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:23.785688 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:23.785785 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:23.785766 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:25.230706 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:25.230679 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:25.231147 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:25.230679 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:25.231147 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:25.230794 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:25.231147 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:25.230815 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:25.231147 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:25.230905 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:25.231147 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:25.230993 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:27.231304 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:27.231278 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:27.231304 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:27.231301 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:27.231785 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:27.231278 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:27.231785 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:27.231384 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc7bk" podUID="d1038ec5-e712-43d1-bf5e-8370bcefd309" Jul 02 18:31:27.231785 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:27.231432 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n98x8" podUID="91c774ed-1026-4a13-91ba-61cb58ae1bca" Jul 02 18:31:27.231785 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:27.231487 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjqpc" podUID="d103adac-ed3a-485a-90c7-9713adec7aba" Jul 02 18:31:28.051991 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.051925 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-23.ec2.internal" event="NodeReady" Jul 02 18:31:28.052106 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.052017 2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 18:31:28.101076 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.101017 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6mmrr"] Jul 02 18:31:28.122301 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.122270 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8gdgj"] Jul 02 18:31:28.122441 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.122399 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.124491 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.124468 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 18:31:28.124627 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.124475 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 18:31:28.124627 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.124526 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 18:31:28.124855 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.124840 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fqsrq\"" Jul 02 18:31:28.124912 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.124894 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 18:31:28.137202 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.137177 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gdgj" Jul 02 18:31:28.137295 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.137180 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gdgj"] Jul 02 18:31:28.137347 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.137296 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6mmrr"] Jul 02 18:31:28.139353 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.139335 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 18:31:28.139472 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.139413 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 18:31:28.139472 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.139436 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2hlqm\"" Jul 02 18:31:28.139601 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.139491 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 18:31:28.195918 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.195899 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4q74j"] Jul 02 18:31:28.209448 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.209431 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4q74j"] Jul 02 18:31:28.209534 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.209509 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.211310 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.211273 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cl9vj\"" Jul 02 18:31:28.211423 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.211378 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 18:31:28.211498 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.211483 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 18:31:28.221840 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.221818 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hldmj\" (UniqueName: \"kubernetes.io/projected/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-kube-api-access-hldmj\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.221923 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.221852 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.221923 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.221882 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-crio-socket\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.221992 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.221942 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cce07b1-d3ba-4353-963f-b3e9f0cf98a4-cert\") pod \"ingress-canary-8gdgj\" (UID: \"6cce07b1-d3ba-4353-963f-b3e9f0cf98a4\") " pod="openshift-ingress-canary/ingress-canary-8gdgj" Jul 02 18:31:28.221992 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.221963 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kltkn\" (UniqueName: \"kubernetes.io/projected/6cce07b1-d3ba-4353-963f-b3e9f0cf98a4-kube-api-access-kltkn\") pod \"ingress-canary-8gdgj\" (UID: \"6cce07b1-d3ba-4353-963f-b3e9f0cf98a4\") " pod="openshift-ingress-canary/ingress-canary-8gdgj" Jul 02 18:31:28.221992 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.221990 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-data-volume\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.222087 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.222004 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.322863 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.322806 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cce07b1-d3ba-4353-963f-b3e9f0cf98a4-cert\") pod \"ingress-canary-8gdgj\" (UID: \"6cce07b1-d3ba-4353-963f-b3e9f0cf98a4\") " pod="openshift-ingress-canary/ingress-canary-8gdgj" Jul 02 18:31:28.322863 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.322830 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kltkn\" (UniqueName: \"kubernetes.io/projected/6cce07b1-d3ba-4353-963f-b3e9f0cf98a4-kube-api-access-kltkn\") pod \"ingress-canary-8gdgj\" (UID: \"6cce07b1-d3ba-4353-963f-b3e9f0cf98a4\") " pod="openshift-ingress-canary/ingress-canary-8gdgj" Jul 02 18:31:28.322863 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.322861 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-data-volume\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.322880 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.322904 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hldmj\" (UniqueName: \"kubernetes.io/projected/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-kube-api-access-hldmj\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.323032 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3de88ed6-612b-4cc2-b74d-648f18596a79-config-volume\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.323072 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de88ed6-612b-4cc2-b74d-648f18596a79-tmp-dir\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.323111 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.323155 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-crio-socket\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.323190 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6lqg\" (UniqueName: \"kubernetes.io/projected/3de88ed6-612b-4cc2-b74d-648f18596a79-kube-api-access-v6lqg\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.323217 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-data-volume\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.323220 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3de88ed6-612b-4cc2-b74d-648f18596a79-metrics-tls\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.323395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.323369 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-crio-socket\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.323773 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.323536 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.326592 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.326568 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.326855 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.326838 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cce07b1-d3ba-4353-963f-b3e9f0cf98a4-cert\") pod \"ingress-canary-8gdgj\" (UID: \"6cce07b1-d3ba-4353-963f-b3e9f0cf98a4\") " pod="openshift-ingress-canary/ingress-canary-8gdgj" Jul 02 18:31:28.329925 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.329910 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kltkn\" (UniqueName: \"kubernetes.io/projected/6cce07b1-d3ba-4353-963f-b3e9f0cf98a4-kube-api-access-kltkn\") pod \"ingress-canary-8gdgj\" (UID: \"6cce07b1-d3ba-4353-963f-b3e9f0cf98a4\") " pod="openshift-ingress-canary/ingress-canary-8gdgj" Jul 02 18:31:28.330042 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.330021 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hldmj\" (UniqueName: \"kubernetes.io/projected/2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e-kube-api-access-hldmj\") pod \"insights-runtime-extractor-6mmrr\" (UID: \"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e\") " pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.424270 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.424244 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3de88ed6-612b-4cc2-b74d-648f18596a79-config-volume\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.424379 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.424276 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de88ed6-612b-4cc2-b74d-648f18596a79-tmp-dir\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.424379 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.424309 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6lqg\" (UniqueName: \"kubernetes.io/projected/3de88ed6-612b-4cc2-b74d-648f18596a79-kube-api-access-v6lqg\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.424379 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.424327 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3de88ed6-612b-4cc2-b74d-648f18596a79-metrics-tls\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.424752 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.424733 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3de88ed6-612b-4cc2-b74d-648f18596a79-config-volume\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.424830 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.424737 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de88ed6-612b-4cc2-b74d-648f18596a79-tmp-dir\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.426344 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.426329 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3de88ed6-612b-4cc2-b74d-648f18596a79-metrics-tls\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.431371 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.431351 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6lqg\" (UniqueName: \"kubernetes.io/projected/3de88ed6-612b-4cc2-b74d-648f18596a79-kube-api-access-v6lqg\") pod \"dns-default-4q74j\" (UID: \"3de88ed6-612b-4cc2-b74d-648f18596a79\") " pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.432253 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.432238 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mmrr" Jul 02 18:31:28.445509 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.445496 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gdgj" Jul 02 18:31:28.517590 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.517565 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:28.626965 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.626783 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6mmrr"] Jul 02 18:31:28.628453 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.628435 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gdgj"] Jul 02 18:31:28.630805 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:28.630778 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4f85cb_78f7_4bcd_9993_8bfa039f7e7e.slice/crio-5a81dea3f781ba3528f7e340474975aa32e226ba5f948d65e175db54c4889e8d WatchSource:0}: Error finding container 5a81dea3f781ba3528f7e340474975aa32e226ba5f948d65e175db54c4889e8d: Status 404 returned error can't find the container with id 5a81dea3f781ba3528f7e340474975aa32e226ba5f948d65e175db54c4889e8d Jul 02 18:31:28.631999 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:28.631979 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cce07b1_d3ba_4353_963f_b3e9f0cf98a4.slice/crio-8a6cde0b8952784b6a7685d10ff0c6c4e49ef464e52e67c8a599d9f2a1494a61 WatchSource:0}: Error finding container 8a6cde0b8952784b6a7685d10ff0c6c4e49ef464e52e67c8a599d9f2a1494a61: Status 404 returned error can't find the container with id 8a6cde0b8952784b6a7685d10ff0c6c4e49ef464e52e67c8a599d9f2a1494a61 Jul 02 18:31:28.679627 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.679601 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4q74j"] Jul 02 18:31:28.683381 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:28.683355 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de88ed6_612b_4cc2_b74d_648f18596a79.slice/crio-0f23382b624285f08297cd4dc8830a3d184076dcc908af6ed00c6f54ebde18fe WatchSource:0}: Error finding container 0f23382b624285f08297cd4dc8830a3d184076dcc908af6ed00c6f54ebde18fe: Status 404 returned error can't find the container with id 0f23382b624285f08297cd4dc8830a3d184076dcc908af6ed00c6f54ebde18fe Jul 02 18:31:28.993797 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:28.993761 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q"] Jul 02 18:31:29.017491 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.017414 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q"] Jul 02 18:31:29.017491 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.017484 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.019569 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.019549 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 18:31:29.020132 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.020007 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 02 18:31:29.020132 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.020015 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 02 18:31:29.020132 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.020026 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 18:31:29.020132 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.020031 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 18:31:29.020132 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.020055 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-srkjj\"" Jul 02 18:31:29.130351 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.130321 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a04623b-23c8-43d9-b06e-4fa9507d4883-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.130472 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.130368 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htnr\" (UniqueName: \"kubernetes.io/projected/0a04623b-23c8-43d9-b06e-4fa9507d4883-kube-api-access-6htnr\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.130535 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.130510 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a04623b-23c8-43d9-b06e-4fa9507d4883-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.130582 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.130551 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a04623b-23c8-43d9-b06e-4fa9507d4883-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.231832 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.231454 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a04623b-23c8-43d9-b06e-4fa9507d4883-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.231832 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.231534 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a04623b-23c8-43d9-b06e-4fa9507d4883-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.231832 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.231582 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a04623b-23c8-43d9-b06e-4fa9507d4883-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.231832 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.231617 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htnr\" (UniqueName: \"kubernetes.io/projected/0a04623b-23c8-43d9-b06e-4fa9507d4883-kube-api-access-6htnr\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.231832 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:29.231739 2552 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 02 18:31:29.231832 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:29.231816 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a04623b-23c8-43d9-b06e-4fa9507d4883-prometheus-operator-tls podName:0a04623b-23c8-43d9-b06e-4fa9507d4883 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:29.731794209 +0000 UTC m=+47.018144030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0a04623b-23c8-43d9-b06e-4fa9507d4883-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-j2z8q" (UID: "0a04623b-23c8-43d9-b06e-4fa9507d4883") : secret "prometheus-operator-tls" not found Jul 02 18:31:29.232448 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.232419 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a04623b-23c8-43d9-b06e-4fa9507d4883-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.235821 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.235678 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:29.235821 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.235737 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:29.235960 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.235942 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a04623b-23c8-43d9-b06e-4fa9507d4883-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.236079 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.235684 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:29.237719 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.237699 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 18:31:29.237979 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.237960 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 18:31:29.238052 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.237982 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pqjw5\"" Jul 02 18:31:29.238052 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.237999 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 18:31:29.238546 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.238307 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xq22w\"" Jul 02 18:31:29.238546 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.238375 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 18:31:29.241197 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.241176 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htnr\" (UniqueName: \"kubernetes.io/projected/0a04623b-23c8-43d9-b06e-4fa9507d4883-kube-api-access-6htnr\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.521031 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.520927 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4q74j" event={"ID":"3de88ed6-612b-4cc2-b74d-648f18596a79","Type":"ContainerStarted","Data":"0f23382b624285f08297cd4dc8830a3d184076dcc908af6ed00c6f54ebde18fe"} Jul 02 18:31:29.522985 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.522953 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gdgj" event={"ID":"6cce07b1-d3ba-4353-963f-b3e9f0cf98a4","Type":"ContainerStarted","Data":"8a6cde0b8952784b6a7685d10ff0c6c4e49ef464e52e67c8a599d9f2a1494a61"} Jul 02 18:31:29.524562 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.524534 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mmrr" event={"ID":"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e","Type":"ContainerStarted","Data":"5a81dea3f781ba3528f7e340474975aa32e226ba5f948d65e175db54c4889e8d"} Jul 02 18:31:29.736041 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.736001 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a04623b-23c8-43d9-b06e-4fa9507d4883-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.741833 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.741802 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a04623b-23c8-43d9-b06e-4fa9507d4883-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j2z8q\" (UID: \"0a04623b-23c8-43d9-b06e-4fa9507d4883\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:29.926066 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:29.926036 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" Jul 02 18:31:30.143946 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.143866 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q"] Jul 02 18:31:30.800158 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:30.800122 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a04623b_23c8_43d9_b06e_4fa9507d4883.slice/crio-aae8db0d0446fccbd01d1bb00bd7b1d925cc0b464c8259171f54025a169b2ae6 WatchSource:0}: Error finding container aae8db0d0446fccbd01d1bb00bd7b1d925cc0b464c8259171f54025a169b2ae6: Status 404 returned error can't find the container with id aae8db0d0446fccbd01d1bb00bd7b1d925cc0b464c8259171f54025a169b2ae6 Jul 02 18:31:30.836301 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.836226 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-866d98c998-dbb2l"] Jul 02 18:31:30.862171 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.862142 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866d98c998-dbb2l"] Jul 02 18:31:30.862312 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.862245 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:30.865108 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.864566 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 18:31:30.865108 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.864608 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 18:31:30.865108 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.864709 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 18:31:30.865108 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.864885 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p67cj\"" Jul 02 18:31:30.865354 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.865195 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 18:31:30.865354 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.865219 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 18:31:30.865354 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.865273 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 18:31:30.865512 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:30.865495 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 18:31:31.046577 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.046547 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-console-config\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.046763 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.046608 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-service-ca\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.046763 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.046651 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2g9l\" (UniqueName: \"kubernetes.io/projected/41baa108-8d52-4588-93bc-5103a04e2043-kube-api-access-m2g9l\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.046855 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.046766 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-oauth-serving-cert\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.046855 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.046817 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-oauth-config\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.046933 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.046870 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-serving-cert\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.147928 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.147849 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-serving-cert\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.147928 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.147896 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-console-config\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.148146 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.148072 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-service-ca\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.148146 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.148111 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2g9l\" (UniqueName: \"kubernetes.io/projected/41baa108-8d52-4588-93bc-5103a04e2043-kube-api-access-m2g9l\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.148596 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.148278 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-oauth-serving-cert\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.148596 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.148345 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-oauth-config\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.148596 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.148537 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-console-config\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.148850 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.148819 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-service-ca\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.149097 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.149071 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-oauth-serving-cert\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.151039 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.151017 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-serving-cert\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.151334 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.151315 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-oauth-config\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.156521 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.156500 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2g9l\" (UniqueName: \"kubernetes.io/projected/41baa108-8d52-4588-93bc-5103a04e2043-kube-api-access-m2g9l\") pod \"console-866d98c998-dbb2l\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.173483 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.173463 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:31.531647 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:31.531595 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" event={"ID":"0a04623b-23c8-43d9-b06e-4fa9507d4883","Type":"ContainerStarted","Data":"aae8db0d0446fccbd01d1bb00bd7b1d925cc0b464c8259171f54025a169b2ae6"} Jul 02 18:31:32.446084 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.444431 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56b4d547bb-nkt66"] Jul 02 18:31:32.456791 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.456764 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.457715 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.457693 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b4d547bb-nkt66"] Jul 02 18:31:32.464387 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.464370 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 18:31:32.559668 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.559616 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-service-ca\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.559848 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.559684 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-serving-cert\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.559848 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.559759 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-oauth-serving-cert\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.559949 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.559853 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-console-config\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.559949 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.559883 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-oauth-config\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.559949 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.559910 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhf2q\" (UniqueName: \"kubernetes.io/projected/7310b827-15a8-4886-b8a6-0ea076b02779-kube-api-access-hhf2q\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.560095 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.560037 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-trusted-ca-bundle\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.661203 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.661171 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-service-ca\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.661381 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.661214 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-serving-cert\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.661381 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.661249 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-oauth-serving-cert\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.661381 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.661283 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-console-config\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.661381 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.661325 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-oauth-config\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.661381 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.661354 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhf2q\" (UniqueName: \"kubernetes.io/projected/7310b827-15a8-4886-b8a6-0ea076b02779-kube-api-access-hhf2q\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.661671 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.661558 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-trusted-ca-bundle\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.661976 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.661946 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-service-ca\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.662101 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.662029 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-oauth-serving-cert\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.662101 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.662091 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-console-config\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.662780 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.662755 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-trusted-ca-bundle\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.664068 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.664045 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-oauth-config\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.664360 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.664335 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-serving-cert\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.668511 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.668489 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhf2q\" (UniqueName: \"kubernetes.io/projected/7310b827-15a8-4886-b8a6-0ea076b02779-kube-api-access-hhf2q\") pod \"console-56b4d547bb-nkt66\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:32.766833 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:32.766756 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:34.667808 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:34.667603 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866d98c998-dbb2l"] Jul 02 18:31:34.671396 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:34.671372 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41baa108_8d52_4588_93bc_5103a04e2043.slice/crio-8c8b416d83fa9910e8e886b73b509397d8b313aa465ef42156cf95838440fc1a WatchSource:0}: Error finding container 8c8b416d83fa9910e8e886b73b509397d8b313aa465ef42156cf95838440fc1a: Status 404 returned error can't find the container with id 8c8b416d83fa9910e8e886b73b509397d8b313aa465ef42156cf95838440fc1a Jul 02 18:31:34.675873 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:34.675851 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b4d547bb-nkt66"] Jul 02 18:31:34.679084 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:34.679063 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7310b827_15a8_4886_b8a6_0ea076b02779.slice/crio-b63f945408e0218a5f24094b0925723b438910ac92ed6e488974f18416cdc0c5 WatchSource:0}: Error finding container b63f945408e0218a5f24094b0925723b438910ac92ed6e488974f18416cdc0c5: Status 404 returned error can't find the container with id b63f945408e0218a5f24094b0925723b438910ac92ed6e488974f18416cdc0c5 Jul 02 18:31:35.542604 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:35.542533 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4d547bb-nkt66" event={"ID":"7310b827-15a8-4886-b8a6-0ea076b02779","Type":"ContainerStarted","Data":"b63f945408e0218a5f24094b0925723b438910ac92ed6e488974f18416cdc0c5"} Jul 02 18:31:35.544258 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:35.544223 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4q74j" event={"ID":"3de88ed6-612b-4cc2-b74d-648f18596a79","Type":"ContainerStarted","Data":"d51894cf05524daf12f07b7278b229ab5447f88a2bf6c76551d41862ed0eda1d"} Jul 02 18:31:35.544402 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:35.544271 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4q74j" event={"ID":"3de88ed6-612b-4cc2-b74d-648f18596a79","Type":"ContainerStarted","Data":"5075dba65c63a3d48358215961b8adcf60bb70d5a0a2087ad01cb44c12967aaf"} Jul 02 18:31:35.544402 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:35.544321 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:35.545675 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:35.545655 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gdgj" event={"ID":"6cce07b1-d3ba-4353-963f-b3e9f0cf98a4","Type":"ContainerStarted","Data":"6ca367b6137af435c810adce09e07bc2e53dfa515bdf2b74f6fd262a60220d23"} Jul 02 18:31:35.546679 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:35.546655 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866d98c998-dbb2l" event={"ID":"41baa108-8d52-4588-93bc-5103a04e2043","Type":"ContainerStarted","Data":"8c8b416d83fa9910e8e886b73b509397d8b313aa465ef42156cf95838440fc1a"} Jul 02 18:31:35.548053 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:35.548032 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mmrr" event={"ID":"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e","Type":"ContainerStarted","Data":"661eb7f8387be79663dde4803b82218d139118a8e0aaa15f7ff3d7cc0777d7ac"} Jul 02 18:31:35.579354 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:35.579316 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4q74j" podStartSLOduration=1.7371556099999999 podStartE2EDuration="7.579304076s" totalImagesPullingTime="5.842148466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:35.564335428 +0000 UTC m=+52.850685285" watchObservedRunningTime="2026-07-02 18:31:35.579304076 +0000 UTC m=+52.865653897" Jul 02 18:31:36.555283 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:36.555065 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mmrr" event={"ID":"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e","Type":"ContainerStarted","Data":"e3d8b975989d0c4db0891ce39b2bcace141a16b9ef06cd7a04d7db0047df4388"} Jul 02 18:31:36.557251 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:36.557215 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" event={"ID":"0a04623b-23c8-43d9-b06e-4fa9507d4883","Type":"ContainerStarted","Data":"738a8145999c4db70e3f8015e4c4891ee4ce2a57df6e2411002cd6a18bfbe5ba"} Jul 02 18:31:36.557378 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:36.557266 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" event={"ID":"0a04623b-23c8-43d9-b06e-4fa9507d4883","Type":"ContainerStarted","Data":"34a6c4499b47e1c087a3d49dc647b637fca59c6570cefc6b5efdc32b53eee349"} Jul 02 18:31:36.577513 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:36.577430 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8gdgj" podStartSLOduration=2.67664512 podStartE2EDuration="8.577414607s" totalImagesPullingTime="5.900769487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:35.579824005 +0000 UTC m=+52.866173859" watchObservedRunningTime="2026-07-02 18:31:36.577414607 +0000 UTC m=+53.863764450" Jul 02 18:31:36.578048 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:36.578017 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j2z8q" podStartSLOduration=3.755628632 podStartE2EDuration="8.578009317s" totalImagesPullingTime="4.822380685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:36.577021212 +0000 UTC m=+53.863371052" watchObservedRunningTime="2026-07-02 18:31:36.578009317 +0000 UTC m=+53.864359158" Jul 02 18:31:40.572933 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:40.572899 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866d98c998-dbb2l" event={"ID":"41baa108-8d52-4588-93bc-5103a04e2043","Type":"ContainerStarted","Data":"93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870"} Jul 02 18:31:40.575037 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:40.575010 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mmrr" event={"ID":"2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e","Type":"ContainerStarted","Data":"f92a7a21ae38de869abc6697f67c2e6b5aaf459cd42830c3f97828ed81796990"} Jul 02 18:31:40.576615 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:40.576595 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4d547bb-nkt66" event={"ID":"7310b827-15a8-4886-b8a6-0ea076b02779","Type":"ContainerStarted","Data":"fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5"} Jul 02 18:31:40.594747 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:40.594701 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-866d98c998-dbb2l" podStartSLOduration=5.707934441 podStartE2EDuration="10.594689096s" totalImagesPullingTime="4.886754655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:40.594099145 +0000 UTC m=+57.880448986" watchObservedRunningTime="2026-07-02 18:31:40.594689096 +0000 UTC m=+57.881038935" Jul 02 18:31:40.615835 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:40.615802 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56b4d547bb-nkt66" podStartSLOduration=3.7432332820000003 podStartE2EDuration="8.615790531s" totalImagesPullingTime="4.872557249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:40.615495043 +0000 UTC m=+57.901844885" watchObservedRunningTime="2026-07-02 18:31:40.615790531 +0000 UTC m=+57.902140373" Jul 02 18:31:40.640490 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:40.640260 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6mmrr" podStartSLOduration=1.9177552530000002 podStartE2EDuration="12.640245805s" totalImagesPullingTime="10.722490552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:31:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:40.639613855 +0000 UTC m=+57.925963696" watchObservedRunningTime="2026-07-02 18:31:40.640245805 +0000 UTC m=+57.926595649" Jul 02 18:31:41.174322 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.174289 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:41.174518 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.174350 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:31:41.175688 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.175659 2552 patch_prober.go:28] interesting pod/console-866d98c998-dbb2l container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 02 18:31:41.175806 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.175734 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-866d98c998-dbb2l" podUID="41baa108-8d52-4588-93bc-5103a04e2043" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 02 18:31:41.458841 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.458769 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759"] Jul 02 18:31:41.461560 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.461542 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.463398 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.463379 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 02 18:31:41.463672 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.463647 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 02 18:31:41.463904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.463889 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pxdq8\"" Jul 02 18:31:41.472550 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.472526 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7"] Jul 02 18:31:41.477630 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.477603 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759"] Jul 02 18:31:41.477864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.477840 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.480696 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.480673 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 02 18:31:41.481153 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.481134 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-49664\"" Jul 02 18:31:41.481493 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.481431 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 02 18:31:41.481493 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.481452 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 02 18:31:41.485297 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.485277 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cwwxs"] Jul 02 18:31:41.487447 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.487431 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.489777 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.489760 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mr82l\"" Jul 02 18:31:41.489872 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.489845 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 18:31:41.489872 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.489860 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 18:31:41.489975 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.489906 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 18:31:41.492540 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.492509 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7"] Jul 02 18:31:41.535999 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.535973 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7c990c6-db5c-4ac4-9599-3a734e3360fe-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.536145 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536024 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-root\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.536145 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536082 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-metrics-client-ca\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.536145 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536107 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/248d42be-542d-44c5-a37c-901bf74745d4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.536145 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536125 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-wtmp\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.536359 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536166 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.536359 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536197 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7c990c6-db5c-4ac4-9599-3a734e3360fe-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.536359 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536224 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7qp\" (UniqueName: \"kubernetes.io/projected/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-api-access-5s7qp\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.536359 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536250 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-tls\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.536359 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536273 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.536359 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536337 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/248d42be-542d-44c5-a37c-901bf74745d4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.536646 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536360 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-sys\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.536646 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536404 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7zk4\" (UniqueName: \"kubernetes.io/projected/248d42be-542d-44c5-a37c-901bf74745d4-kube-api-access-g7zk4\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.536646 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536434 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.536646 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536463 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-textfile\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.536646 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536494 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjbl\" (UniqueName: \"kubernetes.io/projected/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-kube-api-access-4mjbl\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.536646 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536548 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.536646 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536590 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.536646 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.536622 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/248d42be-542d-44c5-a37c-901bf74745d4-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.637213 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637185 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7c990c6-db5c-4ac4-9599-3a734e3360fe-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637271 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-root\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637365 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-metrics-client-ca\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637395 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/248d42be-542d-44c5-a37c-901bf74745d4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637421 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-wtmp\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637466 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637495 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7c990c6-db5c-4ac4-9599-3a734e3360fe-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637536 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f7c990c6-db5c-4ac4-9599-3a734e3360fe-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637539 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7qp\" (UniqueName: \"kubernetes.io/projected/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-api-access-5s7qp\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637582 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-tls\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637609 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637626 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-wtmp\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.637684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637671 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-root\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.637682 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/248d42be-542d-44c5-a37c-901bf74745d4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:41.637779 2552 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:41.637848 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/248d42be-542d-44c5-a37c-901bf74745d4-openshift-state-metrics-tls podName:248d42be-542d-44c5-a37c-901bf74745d4 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:42.137829311 +0000 UTC m=+59.424179148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/248d42be-542d-44c5-a37c-901bf74745d4-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-dl759" (UID: "248d42be-542d-44c5-a37c-901bf74745d4") : secret "openshift-state-metrics-tls" not found Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:41.637908 2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:41.637964 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-tls podName:8e0002e3-0a86-4c4e-bf05-cef319fcd37b nodeName:}" failed. No retries permitted until 2026-07-02 18:31:42.137949838 +0000 UTC m=+59.424299671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-tls") pod "node-exporter-cwwxs" (UID: "8e0002e3-0a86-4c4e-bf05-cef319fcd37b") : secret "node-exporter-tls" not found Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638001 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-sys\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638069 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zk4\" (UniqueName: \"kubernetes.io/projected/248d42be-542d-44c5-a37c-901bf74745d4-kube-api-access-g7zk4\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638076 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-sys\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638102 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638129 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-textfile\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638190 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjbl\" (UniqueName: \"kubernetes.io/projected/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-kube-api-access-4mjbl\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.638268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638198 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7c990c6-db5c-4ac4-9599-3a734e3360fe-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.638873 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638284 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.638873 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638352 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.638873 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638662 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-metrics-client-ca\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.638873 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638722 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-textfile\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.638873 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638727 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.638873 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:41.638820 2552 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 02 18:31:41.638873 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:41.638874 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-tls podName:f7c990c6-db5c-4ac4-9599-3a734e3360fe nodeName:}" failed. No retries permitted until 2026-07-02 18:31:42.138859004 +0000 UTC m=+59.425208841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-twjc7" (UID: "f7c990c6-db5c-4ac4-9599-3a734e3360fe") : secret "kube-state-metrics-tls" not found Jul 02 18:31:41.639206 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.638974 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/248d42be-542d-44c5-a37c-901bf74745d4-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.639206 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.639054 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.639663 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.639621 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/248d42be-542d-44c5-a37c-901bf74745d4-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.641967 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.641945 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.642077 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.642019 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.642077 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.642027 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/248d42be-542d-44c5-a37c-901bf74745d4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:41.648091 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.648058 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7qp\" (UniqueName: \"kubernetes.io/projected/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-api-access-5s7qp\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:41.648220 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.648163 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjbl\" (UniqueName: \"kubernetes.io/projected/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-kube-api-access-4mjbl\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:41.648220 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:41.648176 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zk4\" (UniqueName: \"kubernetes.io/projected/248d42be-542d-44c5-a37c-901bf74745d4-kube-api-access-g7zk4\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:42.142810 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.142772 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-tls\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:42.143005 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.142821 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/248d42be-542d-44c5-a37c-901bf74745d4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:42.143005 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.142885 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:42.145257 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.145227 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e0002e3-0a86-4c4e-bf05-cef319fcd37b-node-exporter-tls\") pod \"node-exporter-cwwxs\" (UID: \"8e0002e3-0a86-4c4e-bf05-cef319fcd37b\") " pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:42.145378 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.145268 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7c990c6-db5c-4ac4-9599-3a734e3360fe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-twjc7\" (UID: \"f7c990c6-db5c-4ac4-9599-3a734e3360fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:42.145445 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.145377 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/248d42be-542d-44c5-a37c-901bf74745d4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-dl759\" (UID: \"248d42be-542d-44c5-a37c-901bf74745d4\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:42.370731 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.370703 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" Jul 02 18:31:42.390092 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.390066 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" Jul 02 18:31:42.397870 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.397811 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cwwxs" Jul 02 18:31:42.408189 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:42.408160 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0002e3_0a86_4c4e_bf05_cef319fcd37b.slice/crio-89443a10e5e6774053aa40560ac1a4f8e431cdd39735fc5e5ecafe9eb08aa6af WatchSource:0}: Error finding container 89443a10e5e6774053aa40560ac1a4f8e431cdd39735fc5e5ecafe9eb08aa6af: Status 404 returned error can't find the container with id 89443a10e5e6774053aa40560ac1a4f8e431cdd39735fc5e5ecafe9eb08aa6af Jul 02 18:31:42.531995 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.531961 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759"] Jul 02 18:31:42.533585 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.533516 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7"] Jul 02 18:31:42.535276 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:42.535218 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248d42be_542d_44c5_a37c_901bf74745d4.slice/crio-5cc18524b2fda6e9fe2053196670e4b248e11458454ecd5c103b871e306f4b0b WatchSource:0}: Error finding container 5cc18524b2fda6e9fe2053196670e4b248e11458454ecd5c103b871e306f4b0b: Status 404 returned error can't find the container with id 5cc18524b2fda6e9fe2053196670e4b248e11458454ecd5c103b871e306f4b0b Jul 02 18:31:42.537072 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:42.537048 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c990c6_db5c_4ac4_9599_3a734e3360fe.slice/crio-d60df205b659ac0c398b9e6d7a2b1e85905f83e2a993e058e20d690336b92e76 WatchSource:0}: Error finding container d60df205b659ac0c398b9e6d7a2b1e85905f83e2a993e058e20d690336b92e76: Status 404 returned error can't find the container with id d60df205b659ac0c398b9e6d7a2b1e85905f83e2a993e058e20d690336b92e76 Jul 02 18:31:42.571057 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.571035 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 18:31:42.575864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.575844 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.579181 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.579160 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 02 18:31:42.579405 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.579386 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 02 18:31:42.579524 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.579433 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 02 18:31:42.579524 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.579503 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 02 18:31:42.579524 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.579520 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 02 18:31:42.579744 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.579725 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vmvd2\"" Jul 02 18:31:42.579828 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.579809 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 02 18:31:42.579924 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.579908 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 02 18:31:42.579997 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.579939 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 02 18:31:42.580161 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.580148 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 02 18:31:42.586376 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.586354 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwwxs" event={"ID":"8e0002e3-0a86-4c4e-bf05-cef319fcd37b","Type":"ContainerStarted","Data":"89443a10e5e6774053aa40560ac1a4f8e431cdd39735fc5e5ecafe9eb08aa6af"} Jul 02 18:31:42.587480 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.587459 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" event={"ID":"248d42be-542d-44c5-a37c-901bf74745d4","Type":"ContainerStarted","Data":"5cc18524b2fda6e9fe2053196670e4b248e11458454ecd5c103b871e306f4b0b"} Jul 02 18:31:42.588446 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.588425 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" event={"ID":"f7c990c6-db5c-4ac4-9599-3a734e3360fe","Type":"ContainerStarted","Data":"d60df205b659ac0c398b9e6d7a2b1e85905f83e2a993e058e20d690336b92e76"} Jul 02 18:31:42.593678 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.592454 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 18:31:42.647555 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647525 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.647949 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647590 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.647949 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647607 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.647949 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647778 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.647949 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647820 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.647949 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647874 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-out\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.647949 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647938 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.648281 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647958 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.648281 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647974 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.648281 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.647992 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.648281 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.648104 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-volume\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.648281 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.648250 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-web-config\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.648483 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.648373 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblv5\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-kube-api-access-cblv5\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.748885 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.748849 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-out\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749074 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.748923 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749074 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.748952 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749074 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.748976 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749074 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.748999 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749074 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749024 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-volume\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749074 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749052 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-web-config\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749074 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749077 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblv5\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-kube-api-access-cblv5\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749121 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749190 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749215 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749241 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749264 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749705 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749593 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.749951 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749827 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.750050 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.749964 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.752263 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.752198 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-volume\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.752263 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.752235 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.752559 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.752526 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-out\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.752820 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.752772 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.753129 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.753097 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.753225 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.753167 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.753225 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.753211 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.754022 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.754003 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-web-config\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.754295 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.754276 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.757743 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.757723 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblv5\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-kube-api-access-cblv5\") pod \"alertmanager-main-0\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:42.767719 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.767676 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:42.767821 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.767788 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:31:42.769492 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.769465 2552 patch_prober.go:28] interesting pod/console-56b4d547bb-nkt66 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.11:8443/health\": dial tcp 10.134.0.11:8443: connect: connection refused" start-of-body= Jul 02 18:31:42.769583 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.769513 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-56b4d547bb-nkt66" podUID="7310b827-15a8-4886-b8a6-0ea076b02779" containerName="console" probeResult="failure" output="Get \"https://10.134.0.11:8443/health\": dial tcp 10.134.0.11:8443: connect: connection refused" Jul 02 18:31:42.887511 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:42.887475 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:31:43.044419 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:43.044188 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 18:31:43.046795 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:43.046765 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f17bb2e_9a19_489e_b0ba_ac280430aef9.slice/crio-4c277ab8279c9d4fc1b436fe36e1b1ea35019ab7067d83fd1d2717c4ff977508 WatchSource:0}: Error finding container 4c277ab8279c9d4fc1b436fe36e1b1ea35019ab7067d83fd1d2717c4ff977508: Status 404 returned error can't find the container with id 4c277ab8279c9d4fc1b436fe36e1b1ea35019ab7067d83fd1d2717c4ff977508 Jul 02 18:31:43.491311 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:43.491282 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-59qz5" Jul 02 18:31:43.592721 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:43.592684 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerStarted","Data":"4c277ab8279c9d4fc1b436fe36e1b1ea35019ab7067d83fd1d2717c4ff977508"} Jul 02 18:31:43.594587 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:43.594560 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" event={"ID":"248d42be-542d-44c5-a37c-901bf74745d4","Type":"ContainerStarted","Data":"92267dfafb842661a9273f8849e721e77717f2ac0043f9fd994c3740fd71c531"} Jul 02 18:31:43.594714 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:43.594608 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" event={"ID":"248d42be-542d-44c5-a37c-901bf74745d4","Type":"ContainerStarted","Data":"6e0dbfea4350de2ed72502c9546ac37bdaa89ad586216164879ad021993adb78"} Jul 02 18:31:44.565521 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.565490 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n"] Jul 02 18:31:44.569217 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.569196 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.573488 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.573019 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 02 18:31:44.573488 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.573069 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 02 18:31:44.573488 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.573321 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 02 18:31:44.577571 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.577456 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 02 18:31:44.578348 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.577968 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 02 18:31:44.578348 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.578065 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-53tdsmvur6gqb\"" Jul 02 18:31:44.578348 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.578186 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gtm77\"" Jul 02 18:31:44.593104 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.593083 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n"] Jul 02 18:31:44.605318 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.605287 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" event={"ID":"248d42be-542d-44c5-a37c-901bf74745d4","Type":"ContainerStarted","Data":"59bb4cba00a30a051461203ee4b0cbeb8a5adbab365dd269174fbc5e977dee9c"} Jul 02 18:31:44.610617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.610540 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" event={"ID":"f7c990c6-db5c-4ac4-9599-3a734e3360fe","Type":"ContainerStarted","Data":"922cd385ec8da84c1425f560b9b55b19cd6c6c5ad98e04ae31cc2514f98c1aaa"} Jul 02 18:31:44.610617 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.610611 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" event={"ID":"f7c990c6-db5c-4ac4-9599-3a734e3360fe","Type":"ContainerStarted","Data":"3bc366a1339e988468f90f387c98f67b1fd017e1ece85e2026edbf366dfc546d"} Jul 02 18:31:44.616661 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.616570 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwwxs" event={"ID":"8e0002e3-0a86-4c4e-bf05-cef319fcd37b","Type":"ContainerStarted","Data":"49b890485783e5e70da320fdd7b2c3422db31584c858dc5cf6b62bf5f59b5a84"} Jul 02 18:31:44.641661 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.641332 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dl759" podStartSLOduration=1.863668923 podStartE2EDuration="3.641312637s" totalImagesPullingTime="1.777643714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:44.633756727 +0000 UTC m=+61.920106572" watchObservedRunningTime="2026-07-02 18:31:44.641312637 +0000 UTC m=+61.927662480" Jul 02 18:31:44.667135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.666771 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.667135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.666846 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.667135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.666931 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-tls\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.667135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.666962 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b172d8bd-adf9-412f-be4a-f47f149ffdfb-metrics-client-ca\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.667135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.666989 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsl9m\" (UniqueName: \"kubernetes.io/projected/b172d8bd-adf9-412f-be4a-f47f149ffdfb-kube-api-access-bsl9m\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.667508 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.667191 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.667508 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.667235 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.667508 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.667258 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-grpc-tls\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.768534 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.768442 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-tls\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.768534 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.768485 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b172d8bd-adf9-412f-be4a-f47f149ffdfb-metrics-client-ca\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.768534 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.768504 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsl9m\" (UniqueName: \"kubernetes.io/projected/b172d8bd-adf9-412f-be4a-f47f149ffdfb-kube-api-access-bsl9m\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.768847 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.768603 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.768847 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.768655 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.768847 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.768687 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-grpc-tls\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.768847 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.768749 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.768847 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.768777 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.769984 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.769928 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b172d8bd-adf9-412f-be4a-f47f149ffdfb-metrics-client-ca\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.771597 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.771568 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.772113 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.772087 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.772209 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.772117 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-grpc-tls\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.772209 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.772127 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.772320 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.772262 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-tls\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.772441 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.772424 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b172d8bd-adf9-412f-be4a-f47f149ffdfb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.780207 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.780160 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsl9m\" (UniqueName: \"kubernetes.io/projected/b172d8bd-adf9-412f-be4a-f47f149ffdfb-kube-api-access-bsl9m\") pod \"thanos-querier-7c9f6559d8-8gg8n\" (UID: \"b172d8bd-adf9-412f-be4a-f47f149ffdfb\") " pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:44.882745 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:44.882705 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:45.052843 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.052805 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n"] Jul 02 18:31:45.181474 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:45.181442 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb172d8bd_adf9_412f_be4a_f47f149ffdfb.slice/crio-a381ae260be18fb7af318ebf4ac9816cfb005ba29a27d6b246660c44859004d8 WatchSource:0}: Error finding container a381ae260be18fb7af318ebf4ac9816cfb005ba29a27d6b246660c44859004d8: Status 404 returned error can't find the container with id a381ae260be18fb7af318ebf4ac9816cfb005ba29a27d6b246660c44859004d8 Jul 02 18:31:45.559243 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.559169 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4q74j" Jul 02 18:31:45.623527 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.623487 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" event={"ID":"f7c990c6-db5c-4ac4-9599-3a734e3360fe","Type":"ContainerStarted","Data":"e5a7f411c711aa873d1936e4e6fed09dc3b0b58d2d8de55a388f576b737052fd"} Jul 02 18:31:45.625023 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.624996 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" event={"ID":"b172d8bd-adf9-412f-be4a-f47f149ffdfb","Type":"ContainerStarted","Data":"a381ae260be18fb7af318ebf4ac9816cfb005ba29a27d6b246660c44859004d8"} Jul 02 18:31:45.626717 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.626693 2552 generic.go:356] "Generic (PLEG): container finished" podID="8e0002e3-0a86-4c4e-bf05-cef319fcd37b" containerID="49b890485783e5e70da320fdd7b2c3422db31584c858dc5cf6b62bf5f59b5a84" exitCode=0 Jul 02 18:31:45.626837 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.626775 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwwxs" event={"ID":"8e0002e3-0a86-4c4e-bf05-cef319fcd37b","Type":"ContainerDied","Data":"49b890485783e5e70da320fdd7b2c3422db31584c858dc5cf6b62bf5f59b5a84"} Jul 02 18:31:45.628953 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.628925 2552 generic.go:356] "Generic (PLEG): container finished" podID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerID="43af107242b30a3e0db47b702a519890464c4d599b3c9c03057acda866a90dd9" exitCode=0 Jul 02 18:31:45.629258 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.629237 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerDied","Data":"43af107242b30a3e0db47b702a519890464c4d599b3c9c03057acda866a90dd9"} Jul 02 18:31:45.643818 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.643777 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-twjc7" podStartSLOduration=2.737504124 podStartE2EDuration="4.643763408s" totalImagesPullingTime="1.906259284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:45.642675631 +0000 UTC m=+62.929025501" watchObservedRunningTime="2026-07-02 18:31:45.643763408 +0000 UTC m=+62.930113308" Jul 02 18:31:45.890704 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.890603 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-654dcbc7d6-69zxg"] Jul 02 18:31:45.893182 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.893163 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:45.895716 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.895522 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 02 18:31:45.895716 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.895555 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 02 18:31:45.895716 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.895570 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 02 18:31:45.895716 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.895628 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 02 18:31:45.895716 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.895528 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-91b5n7k4u3mc5\"" Jul 02 18:31:45.896053 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.895536 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pbr2q\"" Jul 02 18:31:45.906096 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.906055 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-654dcbc7d6-69zxg"] Jul 02 18:31:45.980302 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.980272 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:45.980454 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.980400 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-secret-metrics-server-tls\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:45.980454 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.980444 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-audit-log\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:45.980538 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.980474 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-metrics-server-audit-profiles\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:45.980538 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.980508 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-client-ca-bundle\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:45.980650 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.980553 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwk9\" (UniqueName: \"kubernetes.io/projected/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-kube-api-access-qgwk9\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:45.980650 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:45.980607 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-secret-metrics-server-client-certs\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.081802 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.081766 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-secret-metrics-server-client-certs\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.081802 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.081813 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.082060 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.081908 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-secret-metrics-server-tls\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.082060 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.081947 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-audit-log\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.082060 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.081984 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-metrics-server-audit-profiles\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.082060 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.082024 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-client-ca-bundle\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.082060 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.082056 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwk9\" (UniqueName: \"kubernetes.io/projected/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-kube-api-access-qgwk9\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.082724 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.082663 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-audit-log\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.082895 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.082843 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.083102 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.083076 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-metrics-server-audit-profiles\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.085020 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.084994 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-secret-metrics-server-client-certs\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.085128 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.085093 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-client-ca-bundle\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.085517 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.085497 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-secret-metrics-server-tls\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.089923 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.089901 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwk9\" (UniqueName: \"kubernetes.io/projected/41c322b7-6bea-4cea-8cb5-fe48d49d4d32-kube-api-access-qgwk9\") pod \"metrics-server-654dcbc7d6-69zxg\" (UID: \"41c322b7-6bea-4cea-8cb5-fe48d49d4d32\") " pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.204482 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.204452 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:31:46.272931 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.272901 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt"] Jul 02 18:31:46.275543 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.275523 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:46.277786 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.277761 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2hg2l\"" Jul 02 18:31:46.277913 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.277785 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 02 18:31:46.285342 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.284793 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866d98c998-dbb2l"] Jul 02 18:31:46.298721 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.296804 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt"] Jul 02 18:31:46.331298 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.331265 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dc9dbc5f4-ppq98"] Jul 02 18:31:46.334900 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.334878 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.347129 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.347096 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dc9dbc5f4-ppq98"] Jul 02 18:31:46.364302 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.364277 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-654dcbc7d6-69zxg"] Jul 02 18:31:46.368448 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:46.368419 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c322b7_6bea_4cea_8cb5_fe48d49d4d32.slice/crio-9be70bc79c76e22b74cceb29e036cdf9a6bd4231897be04eef2ed9a4a4680090 WatchSource:0}: Error finding container 9be70bc79c76e22b74cceb29e036cdf9a6bd4231897be04eef2ed9a4a4680090: Status 404 returned error can't find the container with id 9be70bc79c76e22b74cceb29e036cdf9a6bd4231897be04eef2ed9a4a4680090 Jul 02 18:31:46.385897 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.385867 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q528n\" (UniqueName: \"kubernetes.io/projected/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-kube-api-access-q528n\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.385994 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.385915 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-config\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.385994 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.385963 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-service-ca\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.386078 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.385992 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-mhwpt\" (UID: \"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:46.386078 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.386017 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-serving-cert\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.386078 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.386044 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-oauth-serving-cert\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.386211 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.386141 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-oauth-config\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.386211 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.386171 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-trusted-ca-bundle\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.386307 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.386207 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jfm9\" (UniqueName: \"kubernetes.io/projected/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-kube-api-access-9jfm9\") pod \"monitoring-plugin-d5dcff7cf-mhwpt\" (UID: \"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:46.487298 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.487216 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-oauth-config\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.487298 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.487265 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-trusted-ca-bundle\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.487515 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.487301 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jfm9\" (UniqueName: \"kubernetes.io/projected/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-kube-api-access-9jfm9\") pod \"monitoring-plugin-d5dcff7cf-mhwpt\" (UID: \"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:46.487515 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.487340 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q528n\" (UniqueName: \"kubernetes.io/projected/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-kube-api-access-q528n\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.487515 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.487374 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-config\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.487515 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.487397 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-service-ca\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.487515 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.487422 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-mhwpt\" (UID: \"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:46.487515 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.487446 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-serving-cert\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.487515 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.487471 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-oauth-serving-cert\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.487892 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:46.487622 2552 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 02 18:31:46.487892 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:46.487719 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-monitoring-plugin-cert podName:1ad4eb8b-165f-4a95-ab78-1b7c9986bf46 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:46.987696684 +0000 UTC m=+64.274046510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-mhwpt" (UID: "1ad4eb8b-165f-4a95-ab78-1b7c9986bf46") : secret "monitoring-plugin-cert" not found Jul 02 18:31:46.488444 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.488416 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-service-ca\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.488590 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.488555 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-config\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.488680 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.488606 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-trusted-ca-bundle\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.489048 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.489007 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-oauth-serving-cert\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.490130 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.490104 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-oauth-config\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.490360 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.490339 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-serving-cert\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.495231 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.495209 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jfm9\" (UniqueName: \"kubernetes.io/projected/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-kube-api-access-9jfm9\") pod \"monitoring-plugin-d5dcff7cf-mhwpt\" (UID: \"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:46.495383 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.495368 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q528n\" (UniqueName: \"kubernetes.io/projected/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-kube-api-access-q528n\") pod \"console-5dc9dbc5f4-ppq98\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.635600 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.635558 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" event={"ID":"41c322b7-6bea-4cea-8cb5-fe48d49d4d32","Type":"ContainerStarted","Data":"9be70bc79c76e22b74cceb29e036cdf9a6bd4231897be04eef2ed9a4a4680090"} Jul 02 18:31:46.638168 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.638136 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwwxs" event={"ID":"8e0002e3-0a86-4c4e-bf05-cef319fcd37b","Type":"ContainerStarted","Data":"47ffac1952a1f665d7596fb42d3131ce25b91c47715c678ffd547b5f88bc9d3e"} Jul 02 18:31:46.638302 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.638196 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwwxs" event={"ID":"8e0002e3-0a86-4c4e-bf05-cef319fcd37b","Type":"ContainerStarted","Data":"a62a3f999c5b33840acb93eb2f19b657fc78bd5b4f5d6b7a8021d664c64a4d9e"} Jul 02 18:31:46.646778 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.646755 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:46.991123 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:46.991093 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-mhwpt\" (UID: \"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:46.991243 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:46.991199 2552 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 02 18:31:46.991284 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:31:46.991253 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-monitoring-plugin-cert podName:1ad4eb8b-165f-4a95-ab78-1b7c9986bf46 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:47.991239768 +0000 UTC m=+65.277589593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-mhwpt" (UID: "1ad4eb8b-165f-4a95-ab78-1b7c9986bf46") : secret "monitoring-plugin-cert" not found Jul 02 18:31:47.095856 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:47.095770 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cwwxs" podStartSLOduration=3.957384597 podStartE2EDuration="6.095749355s" totalImagesPullingTime="2.03066282s" totalInitContainerRuntime="107.701938ms" isStatefulPod=true podCreationTimestamp="2026-07-02 18:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:46.671356684 +0000 UTC m=+63.957706534" watchObservedRunningTime="2026-07-02 18:31:47.095749355 +0000 UTC m=+64.382099198" Jul 02 18:31:47.096587 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:47.096567 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dc9dbc5f4-ppq98"] Jul 02 18:31:47.100520 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:47.100498 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fe0a4b_9602_4e4a_961a_dba3a3e5f894.slice/crio-b0371e1584d077949521afa0d55babef94f41d78d70c18f8e3fb7b08e5fc0e97 WatchSource:0}: Error finding container b0371e1584d077949521afa0d55babef94f41d78d70c18f8e3fb7b08e5fc0e97: Status 404 returned error can't find the container with id b0371e1584d077949521afa0d55babef94f41d78d70c18f8e3fb7b08e5fc0e97 Jul 02 18:31:47.648913 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:47.646604 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc9dbc5f4-ppq98" event={"ID":"73fe0a4b-9602-4e4a-961a-dba3a3e5f894","Type":"ContainerStarted","Data":"582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266"} Jul 02 18:31:47.648913 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:47.646695 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc9dbc5f4-ppq98" event={"ID":"73fe0a4b-9602-4e4a-961a-dba3a3e5f894","Type":"ContainerStarted","Data":"b0371e1584d077949521afa0d55babef94f41d78d70c18f8e3fb7b08e5fc0e97"} Jul 02 18:31:47.652930 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:47.652439 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" event={"ID":"b172d8bd-adf9-412f-be4a-f47f149ffdfb","Type":"ContainerStarted","Data":"fd949d81dbbddeeec4770f8893d8d9abf32e9c4c84720ea650055200f6b3d4c7"} Jul 02 18:31:47.652930 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:47.652525 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" event={"ID":"b172d8bd-adf9-412f-be4a-f47f149ffdfb","Type":"ContainerStarted","Data":"cb87e31702a88a6133f22b0b9500f0de64b0a3edab9bb51eea1212796ce0308f"} Jul 02 18:31:47.652930 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:47.652571 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" event={"ID":"b172d8bd-adf9-412f-be4a-f47f149ffdfb","Type":"ContainerStarted","Data":"a6688766c1bf55489f9a5e45473ccbee593e337bca3048a4e896afca6528eabf"} Jul 02 18:31:47.670122 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:47.670061 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dc9dbc5f4-ppq98" podStartSLOduration=1.6700377450000001 podStartE2EDuration="1.670037745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:47.668912899 +0000 UTC m=+64.955262743" watchObservedRunningTime="2026-07-02 18:31:47.670037745 +0000 UTC m=+64.956387590" Jul 02 18:31:48.003158 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.003118 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:48.005258 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.003269 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-mhwpt\" (UID: \"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:48.005258 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.003326 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:48.006833 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.006804 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 18:31:48.007142 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.006966 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 18:31:48.007964 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.007934 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ad4eb8b-165f-4a95-ab78-1b7c9986bf46-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-mhwpt\" (UID: \"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:48.018385 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.018361 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1038ec5-e712-43d1-bf5e-8370bcefd309-metrics-certs\") pod \"network-metrics-daemon-qc7bk\" (UID: \"d1038ec5-e712-43d1-bf5e-8370bcefd309\") " pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:48.019141 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.019097 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d103adac-ed3a-485a-90c7-9713adec7aba-original-pull-secret\") pod \"global-pull-secret-syncer-hjqpc\" (UID: \"d103adac-ed3a-485a-90c7-9713adec7aba\") " pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:48.087466 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.087360 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:48.106405 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.106364 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:48.112241 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.109405 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 18:31:48.119097 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.119066 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 18:31:48.132610 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.132580 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhcf\" (UniqueName: \"kubernetes.io/projected/91c774ed-1026-4a13-91ba-61cb58ae1bca-kube-api-access-wlhcf\") pod \"network-check-target-n98x8\" (UID: \"91c774ed-1026-4a13-91ba-61cb58ae1bca\") " pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:48.152439 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.152372 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjqpc" Jul 02 18:31:48.162514 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.162495 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pqjw5\"" Jul 02 18:31:48.169243 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.169222 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xq22w\"" Jul 02 18:31:48.171115 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.171089 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:48.177896 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.177875 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc7bk" Jul 02 18:31:48.661452 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.661313 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" event={"ID":"b172d8bd-adf9-412f-be4a-f47f149ffdfb","Type":"ContainerStarted","Data":"2e5ef499f78ed0511d453c1ce4370b74ba54742dfbd872014d86a237fd0c008d"} Jul 02 18:31:48.663969 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.663603 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" event={"ID":"41c322b7-6bea-4cea-8cb5-fe48d49d4d32","Type":"ContainerStarted","Data":"765ed14b52367990bab185e92ad41583c687dad6f314b5a8ecf0d905fc1beb80"} Jul 02 18:31:48.671736 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.671540 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerStarted","Data":"2f7ad251d241f66eb8ad57faebd8dbee9e1387463d9ccbc94ca2f43a57b527c8"} Jul 02 18:31:48.671736 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.671614 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerStarted","Data":"46fa09594c68bb1312045d5f495043d6e54ea7ca3717ac15bca38474c139a3e2"} Jul 02 18:31:48.680164 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.680140 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qc7bk"] Jul 02 18:31:48.684112 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:48.684076 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1038ec5_e712_43d1_bf5e_8370bcefd309.slice/crio-d18916326324e072decc798e1045869a2742bdea7883eb24bfc52b3b2dd5c01e WatchSource:0}: Error finding container d18916326324e072decc798e1045869a2742bdea7883eb24bfc52b3b2dd5c01e: Status 404 returned error can't find the container with id d18916326324e072decc798e1045869a2742bdea7883eb24bfc52b3b2dd5c01e Jul 02 18:31:48.687616 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.687436 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" podStartSLOduration=1.571295329 podStartE2EDuration="3.687405266s" totalImagesPullingTime="2.116109937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:48.686885671 +0000 UTC m=+65.973235514" watchObservedRunningTime="2026-07-02 18:31:48.687405266 +0000 UTC m=+65.973755113" Jul 02 18:31:48.907731 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.907688 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hjqpc"] Jul 02 18:31:48.910213 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:48.910188 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd103adac_ed3a_485a_90c7_9713adec7aba.slice/crio-1e8a7b7d344fb7d03de15031187a9fe06ac4131b9c0bb5944839141cee3b91eb WatchSource:0}: Error finding container 1e8a7b7d344fb7d03de15031187a9fe06ac4131b9c0bb5944839141cee3b91eb: Status 404 returned error can't find the container with id 1e8a7b7d344fb7d03de15031187a9fe06ac4131b9c0bb5944839141cee3b91eb Jul 02 18:31:48.911844 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.911767 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n98x8"] Jul 02 18:31:48.913894 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:48.913848 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c774ed_1026_4a13_91ba_61cb58ae1bca.slice/crio-8b83f404969cb2709b62c3d7f43082eb37b25ee3bf1632c56a84e9110e8888ef WatchSource:0}: Error finding container 8b83f404969cb2709b62c3d7f43082eb37b25ee3bf1632c56a84e9110e8888ef: Status 404 returned error can't find the container with id 8b83f404969cb2709b62c3d7f43082eb37b25ee3bf1632c56a84e9110e8888ef Jul 02 18:31:48.924107 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:48.924088 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt"] Jul 02 18:31:48.926136 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:48.926113 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad4eb8b_165f_4a95_ab78_1b7c9986bf46.slice/crio-b8abbe8c378801f5f7729e90dea86a8022c42d576ffe4dd4ae7ff5e39f546388 WatchSource:0}: Error finding container b8abbe8c378801f5f7729e90dea86a8022c42d576ffe4dd4ae7ff5e39f546388: Status 404 returned error can't find the container with id b8abbe8c378801f5f7729e90dea86a8022c42d576ffe4dd4ae7ff5e39f546388 Jul 02 18:31:49.679415 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.679340 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" event={"ID":"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46","Type":"ContainerStarted","Data":"b8abbe8c378801f5f7729e90dea86a8022c42d576ffe4dd4ae7ff5e39f546388"} Jul 02 18:31:49.686405 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.686373 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerStarted","Data":"2d21b44a7659f1c2b69442298eee05066d9a09e258a8a92298d5726f96292be8"} Jul 02 18:31:49.686549 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.686448 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerStarted","Data":"b219d6aba209d1e66934e4cec5924f62223a19091dddc91d1ec286303ebd6ef9"} Jul 02 18:31:49.686549 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.686489 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerStarted","Data":"8d3f57f7df7061998fc4973fa60294242b13345ab04acb1bb8986e0ae526e567"} Jul 02 18:31:49.686549 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.686525 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerStarted","Data":"88878121a61fde717c8e2091237fdf1776fa277e6d07debb55180cbdb0680a7c"} Jul 02 18:31:49.689770 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.689701 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qc7bk" event={"ID":"d1038ec5-e712-43d1-bf5e-8370bcefd309","Type":"ContainerStarted","Data":"d18916326324e072decc798e1045869a2742bdea7883eb24bfc52b3b2dd5c01e"} Jul 02 18:31:49.692051 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.691971 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hjqpc" event={"ID":"d103adac-ed3a-485a-90c7-9713adec7aba","Type":"ContainerStarted","Data":"1e8a7b7d344fb7d03de15031187a9fe06ac4131b9c0bb5944839141cee3b91eb"} Jul 02 18:31:49.698712 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.697786 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" event={"ID":"b172d8bd-adf9-412f-be4a-f47f149ffdfb","Type":"ContainerStarted","Data":"bfb9b3d02bdc94483de9e5a212ef97d971ef67cfe4190dded2665e5ae8d161d7"} Jul 02 18:31:49.698712 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.697853 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" event={"ID":"b172d8bd-adf9-412f-be4a-f47f149ffdfb","Type":"ContainerStarted","Data":"68eb03122285fc1f2b264e0a694865e06fc49df18500654ebeeff9e382fe80d2"} Jul 02 18:31:49.698712 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.698626 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:49.700786 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.700717 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n98x8" event={"ID":"91c774ed-1026-4a13-91ba-61cb58ae1bca","Type":"ContainerStarted","Data":"8b83f404969cb2709b62c3d7f43082eb37b25ee3bf1632c56a84e9110e8888ef"} Jul 02 18:31:49.755495 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.754257 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.400769826 podStartE2EDuration="7.754235432s" totalImagesPullingTime="4.337809001s" totalInitContainerRuntime="15.656605ms" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:49.721524466 +0000 UTC m=+67.007874309" watchObservedRunningTime="2026-07-02 18:31:49.754235432 +0000 UTC m=+67.040585275" Jul 02 18:31:49.755495 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:49.755264 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" podStartSLOduration=2.680071532 podStartE2EDuration="5.755251324s" totalImagesPullingTime="3.075179792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:49.753260185 +0000 UTC m=+67.039610027" watchObservedRunningTime="2026-07-02 18:31:49.755251324 +0000 UTC m=+67.041601166" Jul 02 18:31:51.712285 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:51.712247 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" event={"ID":"1ad4eb8b-165f-4a95-ab78-1b7c9986bf46","Type":"ContainerStarted","Data":"be3b23e038fd53ffffb9e3d22fba84c5c54ebdff205bee28d98a13bf84e49c4e"} Jul 02 18:31:51.712755 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:51.712331 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:51.715245 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:51.715195 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qc7bk" event={"ID":"d1038ec5-e712-43d1-bf5e-8370bcefd309","Type":"ContainerStarted","Data":"2a62c9040f6ac4fc2171c6b9f93a280d23585e9b96a5981acde2bf0bc962d90a"} Jul 02 18:31:51.715372 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:51.715268 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qc7bk" event={"ID":"d1038ec5-e712-43d1-bf5e-8370bcefd309","Type":"ContainerStarted","Data":"d0c1689d44d3af8909097a5ee2215663c3cb77fe60f055c5eb283319465d37f6"} Jul 02 18:31:51.717956 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:51.717933 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" Jul 02 18:31:51.729555 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:51.729512 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mhwpt" podStartSLOduration=3.620170263 podStartE2EDuration="5.729493824s" totalImagesPullingTime="2.109323561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:51.729433269 +0000 UTC m=+69.015783109" watchObservedRunningTime="2026-07-02 18:31:51.729493824 +0000 UTC m=+69.015843667" Jul 02 18:31:51.754843 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:51.754791 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qc7bk" podStartSLOduration=66.736897987 podStartE2EDuration="1m8.754768911s" totalImagesPullingTime="2.017870924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:51.752813026 +0000 UTC m=+69.039162870" watchObservedRunningTime="2026-07-02 18:31:51.754768911 +0000 UTC m=+69.041118753" Jul 02 18:31:52.699164 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.699126 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b4d547bb-nkt66"] Jul 02 18:31:52.751584 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.751546 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55bd59f458-ld6fx"] Jul 02 18:31:52.754393 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.754372 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.776500 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.776472 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bd59f458-ld6fx"] Jul 02 18:31:52.852745 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.852343 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-oauth-config\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.852745 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.852421 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-service-ca\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.852745 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.852460 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-serving-cert\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.852745 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.852483 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfvd\" (UniqueName: \"kubernetes.io/projected/d2ee4054-7d43-4a4e-9805-859472549f78-kube-api-access-tzfvd\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.852745 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.852511 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-console-config\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.852745 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.852547 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-trusted-ca-bundle\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.852745 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.852596 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-oauth-serving-cert\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.954115 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.954018 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-oauth-serving-cert\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.954115 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.954087 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-oauth-config\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.954310 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.954142 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-service-ca\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.954310 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.954173 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-serving-cert\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.954310 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.954194 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfvd\" (UniqueName: \"kubernetes.io/projected/d2ee4054-7d43-4a4e-9805-859472549f78-kube-api-access-tzfvd\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.954310 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.954221 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-console-config\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.954310 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.954255 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-trusted-ca-bundle\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.954983 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.954929 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-oauth-serving-cert\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.955121 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.954990 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-service-ca\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.955121 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.955111 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-console-config\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.955336 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.955269 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-trusted-ca-bundle\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.957612 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.957565 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-oauth-config\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.957785 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.957763 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-serving-cert\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:52.963274 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:52.963244 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfvd\" (UniqueName: \"kubernetes.io/projected/d2ee4054-7d43-4a4e-9805-859472549f78-kube-api-access-tzfvd\") pod \"console-55bd59f458-ld6fx\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:53.067997 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:53.067960 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:31:53.836689 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:53.836661 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bd59f458-ld6fx"] Jul 02 18:31:53.839627 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:31:53.839588 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ee4054_7d43_4a4e_9805_859472549f78.slice/crio-1d624ad738302019fc0a2a359a336c7a164e82f951ccf6719a8943cf1bf3165c WatchSource:0}: Error finding container 1d624ad738302019fc0a2a359a336c7a164e82f951ccf6719a8943cf1bf3165c: Status 404 returned error can't find the container with id 1d624ad738302019fc0a2a359a336c7a164e82f951ccf6719a8943cf1bf3165c Jul 02 18:31:54.730932 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:54.730891 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hjqpc" event={"ID":"d103adac-ed3a-485a-90c7-9713adec7aba","Type":"ContainerStarted","Data":"24294b2a6fc2a91835b527abab4ff89e1a0da013e337fbd7abeffba59c9f34fe"} Jul 02 18:31:54.732489 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:54.732463 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd59f458-ld6fx" event={"ID":"d2ee4054-7d43-4a4e-9805-859472549f78","Type":"ContainerStarted","Data":"0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11"} Jul 02 18:31:54.732591 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:54.732509 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd59f458-ld6fx" event={"ID":"d2ee4054-7d43-4a4e-9805-859472549f78","Type":"ContainerStarted","Data":"1d624ad738302019fc0a2a359a336c7a164e82f951ccf6719a8943cf1bf3165c"} Jul 02 18:31:54.733803 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:54.733776 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n98x8" event={"ID":"91c774ed-1026-4a13-91ba-61cb58ae1bca","Type":"ContainerStarted","Data":"a51d76ba20de84cdfd9ad1ffe2fb5e2e90afb9fdf03de032f8a2c7cfe5702def"} Jul 02 18:31:54.733933 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:54.733917 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:31:54.747401 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:54.747349 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hjqpc" podStartSLOduration=66.947855997 podStartE2EDuration="1m11.747331293s" totalImagesPullingTime="4.799475296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:54.746453109 +0000 UTC m=+72.032802949" watchObservedRunningTime="2026-07-02 18:31:54.747331293 +0000 UTC m=+72.033681134" Jul 02 18:31:54.761554 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:54.761504 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n98x8" podStartSLOduration=66.972727762 podStartE2EDuration="1m11.761489188s" totalImagesPullingTime="4.788761426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:54.760389339 +0000 UTC m=+72.046739256" watchObservedRunningTime="2026-07-02 18:31:54.761489188 +0000 UTC m=+72.047839029" Jul 02 18:31:54.778956 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:54.778910 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55bd59f458-ld6fx" podStartSLOduration=2.778893718 podStartE2EDuration="2.778893718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:54.777483246 +0000 UTC m=+72.063833091" watchObservedRunningTime="2026-07-02 18:31:54.778893718 +0000 UTC m=+72.065243562" Jul 02 18:31:56.647739 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:56.647661 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:56.648233 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:56.647798 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:56.652611 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:56.652587 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:31:56.721751 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:56.721725 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c9f6559d8-8gg8n" Jul 02 18:31:56.747197 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:31:56.747164 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:32:03.068815 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:03.068783 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:32:03.069189 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:03.068839 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:32:03.073287 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:03.073264 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:32:03.776028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:03.776001 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:32:03.823595 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:03.823558 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dc9dbc5f4-ppq98"] Jul 02 18:32:06.205266 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:06.205150 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:32:06.205266 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:06.205213 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:32:11.322083 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.322016 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-866d98c998-dbb2l" podUID="41baa108-8d52-4588-93bc-5103a04e2043" containerName="console" containerID="cri-o://93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870" gracePeriod=15 Jul 02 18:32:11.580246 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.580176 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:32:11.729797 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.729764 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-serving-cert\") pod \"41baa108-8d52-4588-93bc-5103a04e2043\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " Jul 02 18:32:11.729797 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.729801 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-console-config\") pod \"41baa108-8d52-4588-93bc-5103a04e2043\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " Jul 02 18:32:11.730028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.729861 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-oauth-serving-cert\") pod \"41baa108-8d52-4588-93bc-5103a04e2043\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " Jul 02 18:32:11.730028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.729926 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-service-ca\") pod \"41baa108-8d52-4588-93bc-5103a04e2043\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " Jul 02 18:32:11.730110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.730060 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2g9l\" (UniqueName: \"kubernetes.io/projected/41baa108-8d52-4588-93bc-5103a04e2043-kube-api-access-m2g9l\") pod \"41baa108-8d52-4588-93bc-5103a04e2043\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " Jul 02 18:32:11.730160 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.730112 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-oauth-config\") pod \"41baa108-8d52-4588-93bc-5103a04e2043\" (UID: \"41baa108-8d52-4588-93bc-5103a04e2043\") " Jul 02 18:32:11.730337 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.730310 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-service-ca" (OuterVolumeSpecName: "service-ca") pod "41baa108-8d52-4588-93bc-5103a04e2043" (UID: "41baa108-8d52-4588-93bc-5103a04e2043"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:11.730421 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.730320 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-console-config" (OuterVolumeSpecName: "console-config") pod "41baa108-8d52-4588-93bc-5103a04e2043" (UID: "41baa108-8d52-4588-93bc-5103a04e2043"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:11.730421 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.730339 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "41baa108-8d52-4588-93bc-5103a04e2043" (UID: "41baa108-8d52-4588-93bc-5103a04e2043"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:11.730421 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.730413 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-console-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:11.730577 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.730432 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-oauth-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:11.730577 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.730443 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41baa108-8d52-4588-93bc-5103a04e2043-service-ca\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:11.732232 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.732202 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "41baa108-8d52-4588-93bc-5103a04e2043" (UID: "41baa108-8d52-4588-93bc-5103a04e2043"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:32:11.732332 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.732291 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41baa108-8d52-4588-93bc-5103a04e2043-kube-api-access-m2g9l" (OuterVolumeSpecName: "kube-api-access-m2g9l") pod "41baa108-8d52-4588-93bc-5103a04e2043" (UID: "41baa108-8d52-4588-93bc-5103a04e2043"). InnerVolumeSpecName "kube-api-access-m2g9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:32:11.732332 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.732307 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "41baa108-8d52-4588-93bc-5103a04e2043" (UID: "41baa108-8d52-4588-93bc-5103a04e2043"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:32:11.803935 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.803898 2552 generic.go:356] "Generic (PLEG): container finished" podID="41baa108-8d52-4588-93bc-5103a04e2043" containerID="93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870" exitCode=0 Jul 02 18:32:11.804125 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.803943 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866d98c998-dbb2l" Jul 02 18:32:11.804125 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.803993 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866d98c998-dbb2l" event={"ID":"41baa108-8d52-4588-93bc-5103a04e2043","Type":"ContainerDied","Data":"93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870"} Jul 02 18:32:11.804125 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.804054 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866d98c998-dbb2l" event={"ID":"41baa108-8d52-4588-93bc-5103a04e2043","Type":"ContainerDied","Data":"8c8b416d83fa9910e8e886b73b509397d8b313aa465ef42156cf95838440fc1a"} Jul 02 18:32:11.804125 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.804082 2552 scope.go:117] "RemoveContainer" containerID="93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870" Jul 02 18:32:11.813029 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.813013 2552 scope.go:117] "RemoveContainer" containerID="93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870" Jul 02 18:32:11.813292 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:32:11.813271 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870\": container with ID starting with 93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870 not found: ID does not exist" containerID="93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870" Jul 02 18:32:11.813361 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.813302 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870"} err="failed to get container status \"93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870\": rpc error: code = NotFound desc = could not find container \"93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870\": container with ID starting with 93b33fc8316d426591f1986d5412299e29d99b1d7e0f3019a7fdd4cfa4f27870 not found: ID does not exist" Jul 02 18:32:11.823752 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.823725 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866d98c998-dbb2l"] Jul 02 18:32:11.827458 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.827435 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-866d98c998-dbb2l"] Jul 02 18:32:11.831257 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.831196 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:11.831257 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.831218 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2g9l\" (UniqueName: \"kubernetes.io/projected/41baa108-8d52-4588-93bc-5103a04e2043-kube-api-access-m2g9l\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:11.831257 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:11.831230 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41baa108-8d52-4588-93bc-5103a04e2043-console-oauth-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:13.237332 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:13.237294 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41baa108-8d52-4588-93bc-5103a04e2043" path="/var/lib/kubelet/pods/41baa108-8d52-4588-93bc-5103a04e2043/volumes" Jul 02 18:32:17.724815 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:17.724776 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-56b4d547bb-nkt66" podUID="7310b827-15a8-4886-b8a6-0ea076b02779" containerName="console" containerID="cri-o://fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5" gracePeriod=15 Jul 02 18:32:17.976719 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:17.976665 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:32:18.088864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.088830 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-console-config\") pod \"7310b827-15a8-4886-b8a6-0ea076b02779\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " Jul 02 18:32:18.089004 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.088876 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-oauth-config\") pod \"7310b827-15a8-4886-b8a6-0ea076b02779\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " Jul 02 18:32:18.089004 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.088912 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhf2q\" (UniqueName: \"kubernetes.io/projected/7310b827-15a8-4886-b8a6-0ea076b02779-kube-api-access-hhf2q\") pod \"7310b827-15a8-4886-b8a6-0ea076b02779\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " Jul 02 18:32:18.089004 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.088948 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-trusted-ca-bundle\") pod \"7310b827-15a8-4886-b8a6-0ea076b02779\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " Jul 02 18:32:18.089004 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.088966 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-service-ca\") pod \"7310b827-15a8-4886-b8a6-0ea076b02779\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " Jul 02 18:32:18.089004 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.089001 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-oauth-serving-cert\") pod \"7310b827-15a8-4886-b8a6-0ea076b02779\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " Jul 02 18:32:18.089270 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.089046 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-serving-cert\") pod \"7310b827-15a8-4886-b8a6-0ea076b02779\" (UID: \"7310b827-15a8-4886-b8a6-0ea076b02779\") " Jul 02 18:32:18.089434 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.089279 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-console-config" (OuterVolumeSpecName: "console-config") pod "7310b827-15a8-4886-b8a6-0ea076b02779" (UID: "7310b827-15a8-4886-b8a6-0ea076b02779"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:18.089536 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.089423 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-service-ca" (OuterVolumeSpecName: "service-ca") pod "7310b827-15a8-4886-b8a6-0ea076b02779" (UID: "7310b827-15a8-4886-b8a6-0ea076b02779"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:18.089536 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.089436 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7310b827-15a8-4886-b8a6-0ea076b02779" (UID: "7310b827-15a8-4886-b8a6-0ea076b02779"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:18.089536 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.089470 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7310b827-15a8-4886-b8a6-0ea076b02779" (UID: "7310b827-15a8-4886-b8a6-0ea076b02779"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:18.091346 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.091324 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7310b827-15a8-4886-b8a6-0ea076b02779" (UID: "7310b827-15a8-4886-b8a6-0ea076b02779"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:32:18.091550 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.091530 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7310b827-15a8-4886-b8a6-0ea076b02779-kube-api-access-hhf2q" (OuterVolumeSpecName: "kube-api-access-hhf2q") pod "7310b827-15a8-4886-b8a6-0ea076b02779" (UID: "7310b827-15a8-4886-b8a6-0ea076b02779"). InnerVolumeSpecName "kube-api-access-hhf2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:32:18.091608 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.091550 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7310b827-15a8-4886-b8a6-0ea076b02779" (UID: "7310b827-15a8-4886-b8a6-0ea076b02779"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:32:18.190216 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.190163 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-oauth-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:18.190216 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.190207 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhf2q\" (UniqueName: \"kubernetes.io/projected/7310b827-15a8-4886-b8a6-0ea076b02779-kube-api-access-hhf2q\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:18.190216 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.190223 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-trusted-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:18.190452 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.190235 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-service-ca\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:18.190452 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.190247 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-oauth-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:18.190452 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.190258 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7310b827-15a8-4886-b8a6-0ea076b02779-console-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:18.190452 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.190269 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7310b827-15a8-4886-b8a6-0ea076b02779-console-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:18.840010 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.839968 2552 generic.go:356] "Generic (PLEG): container finished" podID="7310b827-15a8-4886-b8a6-0ea076b02779" containerID="fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5" exitCode=0 Jul 02 18:32:18.840415 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.840032 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4d547bb-nkt66" Jul 02 18:32:18.840415 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.840052 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4d547bb-nkt66" event={"ID":"7310b827-15a8-4886-b8a6-0ea076b02779","Type":"ContainerDied","Data":"fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5"} Jul 02 18:32:18.840415 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.840129 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4d547bb-nkt66" event={"ID":"7310b827-15a8-4886-b8a6-0ea076b02779","Type":"ContainerDied","Data":"b63f945408e0218a5f24094b0925723b438910ac92ed6e488974f18416cdc0c5"} Jul 02 18:32:18.840415 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.840150 2552 scope.go:117] "RemoveContainer" containerID="fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5" Jul 02 18:32:18.849270 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.849248 2552 scope.go:117] "RemoveContainer" containerID="fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5" Jul 02 18:32:18.849510 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:32:18.849491 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5\": container with ID starting with fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5 not found: ID does not exist" containerID="fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5" Jul 02 18:32:18.849557 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.849519 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5"} err="failed to get container status \"fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5\": rpc error: code = NotFound desc = could not find container \"fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5\": container with ID starting with fca7f11f8d5b830b6c2a5f3666f6771ed07726ad769df9dcc8ce580166d302c5 not found: ID does not exist" Jul 02 18:32:18.860183 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.860157 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b4d547bb-nkt66"] Jul 02 18:32:18.864260 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:18.864236 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56b4d547bb-nkt66"] Jul 02 18:32:19.235134 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:19.235100 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7310b827-15a8-4886-b8a6-0ea076b02779" path="/var/lib/kubelet/pods/7310b827-15a8-4886-b8a6-0ea076b02779/volumes" Jul 02 18:32:25.740525 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:25.740491 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n98x8" Jul 02 18:32:26.209859 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:26.209831 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:32:26.213940 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:26.213904 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-654dcbc7d6-69zxg" Jul 02 18:32:28.844680 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:28.844603 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5dc9dbc5f4-ppq98" podUID="73fe0a4b-9602-4e4a-961a-dba3a3e5f894" containerName="console" containerID="cri-o://582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266" gracePeriod=15 Jul 02 18:32:29.099714 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.099656 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:32:29.169472 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.169435 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-serving-cert\") pod \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " Jul 02 18:32:29.169682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.169501 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q528n\" (UniqueName: \"kubernetes.io/projected/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-kube-api-access-q528n\") pod \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " Jul 02 18:32:29.169682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.169527 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-oauth-serving-cert\") pod \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " Jul 02 18:32:29.169682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.169560 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-service-ca\") pod \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " Jul 02 18:32:29.169682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.169600 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-trusted-ca-bundle\") pod \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " Jul 02 18:32:29.169682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.169627 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-config\") pod \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " Jul 02 18:32:29.169957 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.169703 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-oauth-config\") pod \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\" (UID: \"73fe0a4b-9602-4e4a-961a-dba3a3e5f894\") " Jul 02 18:32:29.170007 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.169987 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "73fe0a4b-9602-4e4a-961a-dba3a3e5f894" (UID: "73fe0a4b-9602-4e4a-961a-dba3a3e5f894"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:29.170062 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.170023 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-service-ca" (OuterVolumeSpecName: "service-ca") pod "73fe0a4b-9602-4e4a-961a-dba3a3e5f894" (UID: "73fe0a4b-9602-4e4a-961a-dba3a3e5f894"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:29.170107 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.170074 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "73fe0a4b-9602-4e4a-961a-dba3a3e5f894" (UID: "73fe0a4b-9602-4e4a-961a-dba3a3e5f894"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:29.170107 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.170090 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-config" (OuterVolumeSpecName: "console-config") pod "73fe0a4b-9602-4e4a-961a-dba3a3e5f894" (UID: "73fe0a4b-9602-4e4a-961a-dba3a3e5f894"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:32:29.171871 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.171843 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "73fe0a4b-9602-4e4a-961a-dba3a3e5f894" (UID: "73fe0a4b-9602-4e4a-961a-dba3a3e5f894"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:32:29.171871 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.171857 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-kube-api-access-q528n" (OuterVolumeSpecName: "kube-api-access-q528n") pod "73fe0a4b-9602-4e4a-961a-dba3a3e5f894" (UID: "73fe0a4b-9602-4e4a-961a-dba3a3e5f894"). InnerVolumeSpecName "kube-api-access-q528n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:32:29.172007 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.171907 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "73fe0a4b-9602-4e4a-961a-dba3a3e5f894" (UID: "73fe0a4b-9602-4e4a-961a-dba3a3e5f894"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:32:29.271125 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.271090 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-service-ca\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:29.271125 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.271118 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-trusted-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:29.271125 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.271129 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:29.271339 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.271143 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-oauth-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:29.271339 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.271153 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-console-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:29.271339 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.271162 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q528n\" (UniqueName: \"kubernetes.io/projected/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-kube-api-access-q528n\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:29.271339 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.271171 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73fe0a4b-9602-4e4a-961a-dba3a3e5f894-oauth-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:32:29.887482 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.887451 2552 generic.go:356] "Generic (PLEG): container finished" podID="73fe0a4b-9602-4e4a-961a-dba3a3e5f894" containerID="582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266" exitCode=0 Jul 02 18:32:29.888015 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.887507 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc9dbc5f4-ppq98" Jul 02 18:32:29.888015 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.887547 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc9dbc5f4-ppq98" event={"ID":"73fe0a4b-9602-4e4a-961a-dba3a3e5f894","Type":"ContainerDied","Data":"582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266"} Jul 02 18:32:29.888015 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.887602 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc9dbc5f4-ppq98" event={"ID":"73fe0a4b-9602-4e4a-961a-dba3a3e5f894","Type":"ContainerDied","Data":"b0371e1584d077949521afa0d55babef94f41d78d70c18f8e3fb7b08e5fc0e97"} Jul 02 18:32:29.888015 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.887625 2552 scope.go:117] "RemoveContainer" containerID="582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266" Jul 02 18:32:29.895738 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.895714 2552 scope.go:117] "RemoveContainer" containerID="582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266" Jul 02 18:32:29.895985 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:32:29.895966 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266\": container with ID starting with 582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266 not found: ID does not exist" containerID="582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266" Jul 02 18:32:29.896045 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.895993 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266"} err="failed to get container status \"582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266\": rpc error: code = NotFound desc = could not find container \"582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266\": container with ID starting with 582cd87ec00744bcff747c496fda5fc108c4ee07872424a637eedf734b44d266 not found: ID does not exist" Jul 02 18:32:29.901562 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.901541 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dc9dbc5f4-ppq98"] Jul 02 18:32:29.904805 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:29.904787 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dc9dbc5f4-ppq98"] Jul 02 18:32:31.235371 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:32:31.235340 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73fe0a4b-9602-4e4a-961a-dba3a3e5f894" path="/var/lib/kubelet/pods/73fe0a4b-9602-4e4a-961a-dba3a3e5f894/volumes" Jul 02 18:33:01.839292 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:01.839254 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 18:33:01.840086 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:01.839565 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="alertmanager" containerID="cri-o://46fa09594c68bb1312045d5f495043d6e54ea7ca3717ac15bca38474c139a3e2" gracePeriod=120 Jul 02 18:33:01.840086 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:01.839624 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy-metric" containerID="cri-o://b219d6aba209d1e66934e4cec5924f62223a19091dddc91d1ec286303ebd6ef9" gracePeriod=120 Jul 02 18:33:01.840086 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:01.839708 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="prom-label-proxy" containerID="cri-o://2d21b44a7659f1c2b69442298eee05066d9a09e258a8a92298d5726f96292be8" gracePeriod=120 Jul 02 18:33:01.840086 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:01.839756 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="config-reloader" containerID="cri-o://2f7ad251d241f66eb8ad57faebd8dbee9e1387463d9ccbc94ca2f43a57b527c8" gracePeriod=120 Jul 02 18:33:01.840086 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:01.839711 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy-web" containerID="cri-o://88878121a61fde717c8e2091237fdf1776fa277e6d07debb55180cbdb0680a7c" gracePeriod=120 Jul 02 18:33:01.840086 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:01.839740 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy" containerID="cri-o://8d3f57f7df7061998fc4973fa60294242b13345ab04acb1bb8986e0ae526e567" gracePeriod=120 Jul 02 18:33:02.010239 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:02.010204 2552 generic.go:356] "Generic (PLEG): container finished" podID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerID="2d21b44a7659f1c2b69442298eee05066d9a09e258a8a92298d5726f96292be8" exitCode=0 Jul 02 18:33:02.010239 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:02.010228 2552 generic.go:356] "Generic (PLEG): container finished" podID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerID="8d3f57f7df7061998fc4973fa60294242b13345ab04acb1bb8986e0ae526e567" exitCode=0 Jul 02 18:33:02.010239 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:02.010238 2552 generic.go:356] "Generic (PLEG): container finished" podID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerID="2f7ad251d241f66eb8ad57faebd8dbee9e1387463d9ccbc94ca2f43a57b527c8" exitCode=0 Jul 02 18:33:02.010239 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:02.010247 2552 generic.go:356] "Generic (PLEG): container finished" podID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerID="46fa09594c68bb1312045d5f495043d6e54ea7ca3717ac15bca38474c139a3e2" exitCode=0 Jul 02 18:33:02.010484 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:02.010275 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerDied","Data":"2d21b44a7659f1c2b69442298eee05066d9a09e258a8a92298d5726f96292be8"} Jul 02 18:33:02.010484 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:02.010363 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerDied","Data":"8d3f57f7df7061998fc4973fa60294242b13345ab04acb1bb8986e0ae526e567"} Jul 02 18:33:02.010484 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:02.010397 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerDied","Data":"2f7ad251d241f66eb8ad57faebd8dbee9e1387463d9ccbc94ca2f43a57b527c8"} Jul 02 18:33:02.010484 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:02.010422 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerDied","Data":"46fa09594c68bb1312045d5f495043d6e54ea7ca3717ac15bca38474c139a3e2"} Jul 02 18:33:03.018835 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.018807 2552 generic.go:356] "Generic (PLEG): container finished" podID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerID="b219d6aba209d1e66934e4cec5924f62223a19091dddc91d1ec286303ebd6ef9" exitCode=0 Jul 02 18:33:03.018835 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.018831 2552 generic.go:356] "Generic (PLEG): container finished" podID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerID="88878121a61fde717c8e2091237fdf1776fa277e6d07debb55180cbdb0680a7c" exitCode=0 Jul 02 18:33:03.019207 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.018877 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerDied","Data":"b219d6aba209d1e66934e4cec5924f62223a19091dddc91d1ec286303ebd6ef9"} Jul 02 18:33:03.019207 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.018947 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerDied","Data":"88878121a61fde717c8e2091237fdf1776fa277e6d07debb55180cbdb0680a7c"} Jul 02 18:33:03.086997 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.086967 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:03.144135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144050 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-tls-assets\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144080 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144104 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-metrics-client-ca\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144135 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144129 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-trusted-ca-bundle\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144435 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144162 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-main-db\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144435 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144299 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-web-config\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144435 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144356 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-volume\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144435 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144390 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-web\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144435 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144418 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144726 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144459 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-cluster-tls-config\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144726 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144494 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-out\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144726 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144575 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-main-tls\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144726 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144621 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cblv5\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-kube-api-access-cblv5\") pod \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\" (UID: \"4f17bb2e-9a19-489e-b0ba-ac280430aef9\") " Jul 02 18:33:03.144922 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.144616 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:33:03.145287 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.145105 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:33:03.145287 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.145128 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 18:33:03.147523 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.147484 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:33:03.148391 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.148360 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:33:03.148858 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.148817 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-out" (OuterVolumeSpecName: "config-out") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 18:33:03.149047 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.149023 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-kube-api-access-cblv5" (OuterVolumeSpecName: "kube-api-access-cblv5") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "kube-api-access-cblv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:33:03.149137 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.149029 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:33:03.149137 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.149089 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:33:03.149277 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.149251 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-volume" (OuterVolumeSpecName: "config-volume") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:33:03.149530 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.149508 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:33:03.153288 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.153263 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:33:03.158926 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.158905 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-web-config" (OuterVolumeSpecName: "web-config") pod "4f17bb2e-9a19-489e-b0ba-ac280430aef9" (UID: "4f17bb2e-9a19-489e-b0ba-ac280430aef9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:33:03.245968 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.245943 2552 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-cluster-tls-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.245968 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.245967 2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-out\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.245976 2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-main-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.245986 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblv5\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-kube-api-access-cblv5\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.245995 2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f17bb2e-9a19-489e-b0ba-ac280430aef9-tls-assets\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.246005 2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.246014 2552 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-metrics-client-ca\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.246023 2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.246032 2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f17bb2e-9a19-489e-b0ba-ac280430aef9-alertmanager-main-db\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.246039 2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-web-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.246049 2552 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-config-volume\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.246057 2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:03.246110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:03.246065 2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f17bb2e-9a19-489e-b0ba-ac280430aef9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:04.025419 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.025386 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f17bb2e-9a19-489e-b0ba-ac280430aef9","Type":"ContainerDied","Data":"4c277ab8279c9d4fc1b436fe36e1b1ea35019ab7067d83fd1d2717c4ff977508"} Jul 02 18:33:04.025842 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.025434 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.025842 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.025462 2552 scope.go:117] "RemoveContainer" containerID="2d21b44a7659f1c2b69442298eee05066d9a09e258a8a92298d5726f96292be8" Jul 02 18:33:04.038242 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.038208 2552 scope.go:117] "RemoveContainer" containerID="b219d6aba209d1e66934e4cec5924f62223a19091dddc91d1ec286303ebd6ef9" Jul 02 18:33:04.045835 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.045808 2552 scope.go:117] "RemoveContainer" containerID="8d3f57f7df7061998fc4973fa60294242b13345ab04acb1bb8986e0ae526e567" Jul 02 18:33:04.047488 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.047464 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 18:33:04.052037 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.052010 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 18:33:04.054404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.054382 2552 scope.go:117] "RemoveContainer" containerID="88878121a61fde717c8e2091237fdf1776fa277e6d07debb55180cbdb0680a7c" Jul 02 18:33:04.061281 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.061262 2552 scope.go:117] "RemoveContainer" containerID="2f7ad251d241f66eb8ad57faebd8dbee9e1387463d9ccbc94ca2f43a57b527c8" Jul 02 18:33:04.068450 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.068429 2552 scope.go:117] "RemoveContainer" containerID="46fa09594c68bb1312045d5f495043d6e54ea7ca3717ac15bca38474c139a3e2" Jul 02 18:33:04.075965 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.075940 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 18:33:04.076271 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076243 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="init-config-reloader" Jul 02 18:33:04.076271 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076261 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="init-config-reloader" Jul 02 18:33:04.076271 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076274 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="alertmanager" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076281 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="alertmanager" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076287 2552 scope.go:117] "RemoveContainer" containerID="43af107242b30a3e0db47b702a519890464c4d599b3c9c03057acda866a90dd9" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076294 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy-web" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076375 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy-web" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076394 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy-metric" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076405 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy-metric" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076419 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="config-reloader" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076428 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="config-reloader" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076460 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy" Jul 02 18:33:04.076476 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076470 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076491 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41baa108-8d52-4588-93bc-5103a04e2043" containerName="console" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076501 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="41baa108-8d52-4588-93bc-5103a04e2043" containerName="console" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076511 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="prom-label-proxy" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076520 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="prom-label-proxy" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076535 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73fe0a4b-9602-4e4a-961a-dba3a3e5f894" containerName="console" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076543 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73fe0a4b-9602-4e4a-961a-dba3a3e5f894" containerName="console" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076554 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7310b827-15a8-4886-b8a6-0ea076b02779" containerName="console" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076563 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7310b827-15a8-4886-b8a6-0ea076b02779" containerName="console" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076678 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="alertmanager" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076693 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy-metric" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076702 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73fe0a4b-9602-4e4a-961a-dba3a3e5f894" containerName="console" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076712 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076720 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7310b827-15a8-4886-b8a6-0ea076b02779" containerName="console" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076728 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="config-reloader" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076737 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="prom-label-proxy" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076747 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" containerName="kube-rbac-proxy-web" Jul 02 18:33:04.076955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.076757 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="41baa108-8d52-4588-93bc-5103a04e2043" containerName="console" Jul 02 18:33:04.080704 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.080682 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.082971 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.082942 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 02 18:33:04.083100 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.083007 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 02 18:33:04.083100 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.083034 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 02 18:33:04.083100 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.083041 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 02 18:33:04.083264 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.083203 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 02 18:33:04.083264 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.083257 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 02 18:33:04.083376 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.083356 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 02 18:33:04.083434 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.083398 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 02 18:33:04.083518 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.083500 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vmvd2\"" Jul 02 18:33:04.089428 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.089408 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 02 18:33:04.093122 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.093094 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 18:33:04.155710 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155589 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.155710 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155662 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cae0007-99f0-4e1d-a521-97ce445173d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.155710 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155712 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155782 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155819 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6cae0007-99f0-4e1d-a521-97ce445173d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155845 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cae0007-99f0-4e1d-a521-97ce445173d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155872 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155897 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cae0007-99f0-4e1d-a521-97ce445173d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155925 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xlwg\" (UniqueName: \"kubernetes.io/projected/6cae0007-99f0-4e1d-a521-97ce445173d0-kube-api-access-2xlwg\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155982 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cae0007-99f0-4e1d-a521-97ce445173d0-config-out\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.155999 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.156013 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-web-config\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.156304 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.156094 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257011 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.256973 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257194 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257035 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cae0007-99f0-4e1d-a521-97ce445173d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257194 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257072 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257194 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257107 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257194 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257129 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6cae0007-99f0-4e1d-a521-97ce445173d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257194 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257150 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cae0007-99f0-4e1d-a521-97ce445173d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257194 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257175 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257518 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257200 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cae0007-99f0-4e1d-a521-97ce445173d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257518 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257232 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xlwg\" (UniqueName: \"kubernetes.io/projected/6cae0007-99f0-4e1d-a521-97ce445173d0-kube-api-access-2xlwg\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257518 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257293 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cae0007-99f0-4e1d-a521-97ce445173d0-config-out\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257518 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257319 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257518 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257342 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-web-config\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257518 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257380 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.257834 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.257576 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6cae0007-99f0-4e1d-a521-97ce445173d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.258042 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.258020 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cae0007-99f0-4e1d-a521-97ce445173d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.258341 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.258314 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cae0007-99f0-4e1d-a521-97ce445173d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.260192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.260162 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.260290 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.260252 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.260350 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.260289 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.260350 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.260337 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.260836 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.260810 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.260934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.260862 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cae0007-99f0-4e1d-a521-97ce445173d0-config-out\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.260934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.260843 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.261114 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.261094 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cae0007-99f0-4e1d-a521-97ce445173d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.262567 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.262546 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cae0007-99f0-4e1d-a521-97ce445173d0-web-config\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.267979 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.267956 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xlwg\" (UniqueName: \"kubernetes.io/projected/6cae0007-99f0-4e1d-a521-97ce445173d0-kube-api-access-2xlwg\") pod \"alertmanager-main-0\" (UID: \"6cae0007-99f0-4e1d-a521-97ce445173d0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.394062 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.394030 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 18:33:04.529803 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:04.529769 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 18:33:04.532581 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:33:04.532550 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cae0007_99f0_4e1d_a521_97ce445173d0.slice/crio-70414c76a38a5594c40038548083d01b8d28f7b7c24bc35d67716d96085146f2 WatchSource:0}: Error finding container 70414c76a38a5594c40038548083d01b8d28f7b7c24bc35d67716d96085146f2: Status 404 returned error can't find the container with id 70414c76a38a5594c40038548083d01b8d28f7b7c24bc35d67716d96085146f2 Jul 02 18:33:05.031403 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:05.031371 2552 generic.go:356] "Generic (PLEG): container finished" podID="6cae0007-99f0-4e1d-a521-97ce445173d0" containerID="b175566f17ea086f84c1acef1ac9a517face8b14a49610cc94614693a7700e57" exitCode=0 Jul 02 18:33:05.031817 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:05.031435 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6cae0007-99f0-4e1d-a521-97ce445173d0","Type":"ContainerDied","Data":"b175566f17ea086f84c1acef1ac9a517face8b14a49610cc94614693a7700e57"} Jul 02 18:33:05.031817 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:05.031483 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6cae0007-99f0-4e1d-a521-97ce445173d0","Type":"ContainerStarted","Data":"70414c76a38a5594c40038548083d01b8d28f7b7c24bc35d67716d96085146f2"} Jul 02 18:33:05.236569 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:05.236541 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f17bb2e-9a19-489e-b0ba-ac280430aef9" path="/var/lib/kubelet/pods/4f17bb2e-9a19-489e-b0ba-ac280430aef9/volumes" Jul 02 18:33:06.039249 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:06.039206 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6cae0007-99f0-4e1d-a521-97ce445173d0","Type":"ContainerStarted","Data":"0bac2a75759b968a37b0360b5d231ca6dee74551ec9b04ad277625ad56d3bd8e"} Jul 02 18:33:06.039682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:06.039330 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6cae0007-99f0-4e1d-a521-97ce445173d0","Type":"ContainerStarted","Data":"139f2be85028df431c2d276c28732f8490c22752edc427cfc4ef312e89eff755"} Jul 02 18:33:06.039682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:06.039374 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6cae0007-99f0-4e1d-a521-97ce445173d0","Type":"ContainerStarted","Data":"73f127488bd7258e808c4152121045d1bf767e6bb144251a576db153e9907c54"} Jul 02 18:33:06.039682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:06.039409 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6cae0007-99f0-4e1d-a521-97ce445173d0","Type":"ContainerStarted","Data":"9c6e880e63d6eb2531a28a5863c7f1e4ebacf934cee5d904b274f0716e5f7a1e"} Jul 02 18:33:06.039682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:06.039448 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6cae0007-99f0-4e1d-a521-97ce445173d0","Type":"ContainerStarted","Data":"ca8e4239150ac5fa137f7487c8e45dc8bb71fbcf7072f49a6f8d80c2210f54c1"} Jul 02 18:33:06.039682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:06.039490 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6cae0007-99f0-4e1d-a521-97ce445173d0","Type":"ContainerStarted","Data":"3ad5a82b032a581610418a897e1880f1ed01991c877fbe62791e80c1ac1013d3"} Jul 02 18:33:06.062033 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:06.061975 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.043392578 podStartE2EDuration="2.061958573s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.565995ms" isStatefulPod=false podCreationTimestamp="2026-07-02 18:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:33:06.061174081 +0000 UTC m=+143.347523926" watchObservedRunningTime="2026-07-02 18:33:06.061958573 +0000 UTC m=+143.348308620" Jul 02 18:33:12.326868 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.326835 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57dd756d5f-2tjpw"] Jul 02 18:33:12.329270 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.329233 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.342212 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.342184 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57dd756d5f-2tjpw"] Jul 02 18:33:12.425423 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.425390 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-serving-cert\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.425423 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.425425 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-oauth-config\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.425590 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.425451 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-config\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.425590 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.425533 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-trusted-ca-bundle\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.425590 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.425573 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-service-ca\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.425729 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.425624 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrf7\" (UniqueName: \"kubernetes.io/projected/e8c5a11f-e665-4692-9262-a1fcb71f9104-kube-api-access-fkrf7\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.425729 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.425685 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-oauth-serving-cert\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.526592 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.526551 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-service-ca\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.526819 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.526605 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrf7\" (UniqueName: \"kubernetes.io/projected/e8c5a11f-e665-4692-9262-a1fcb71f9104-kube-api-access-fkrf7\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.526819 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.526664 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-oauth-serving-cert\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.526819 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.526686 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-serving-cert\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.526819 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.526702 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-oauth-config\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.526819 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.526726 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-config\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.526819 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.526750 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-trusted-ca-bundle\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.527436 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.527405 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-service-ca\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.527567 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.527543 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-oauth-serving-cert\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.527629 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.527543 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-config\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.527709 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.527667 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-trusted-ca-bundle\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.529661 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.529620 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-serving-cert\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.529661 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.529650 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-oauth-config\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.533581 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.533549 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrf7\" (UniqueName: \"kubernetes.io/projected/e8c5a11f-e665-4692-9262-a1fcb71f9104-kube-api-access-fkrf7\") pod \"console-57dd756d5f-2tjpw\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.638733 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.638612 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:12.769767 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:12.769737 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57dd756d5f-2tjpw"] Jul 02 18:33:12.772800 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:33:12.772762 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c5a11f_e665_4692_9262_a1fcb71f9104.slice/crio-1f5cb0ac05d2cd3a60ab4481f8e4d1c484f4944299a9eeb238f1d114d1dbbddb WatchSource:0}: Error finding container 1f5cb0ac05d2cd3a60ab4481f8e4d1c484f4944299a9eeb238f1d114d1dbbddb: Status 404 returned error can't find the container with id 1f5cb0ac05d2cd3a60ab4481f8e4d1c484f4944299a9eeb238f1d114d1dbbddb Jul 02 18:33:13.070347 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:13.070307 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57dd756d5f-2tjpw" event={"ID":"e8c5a11f-e665-4692-9262-a1fcb71f9104","Type":"ContainerStarted","Data":"9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565"} Jul 02 18:33:13.070523 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:13.070364 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57dd756d5f-2tjpw" event={"ID":"e8c5a11f-e665-4692-9262-a1fcb71f9104","Type":"ContainerStarted","Data":"1f5cb0ac05d2cd3a60ab4481f8e4d1c484f4944299a9eeb238f1d114d1dbbddb"} Jul 02 18:33:13.089484 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:13.089438 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57dd756d5f-2tjpw" podStartSLOduration=1.089420527 podStartE2EDuration="1.089420527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:33:13.087799206 +0000 UTC m=+150.374149049" watchObservedRunningTime="2026-07-02 18:33:13.089420527 +0000 UTC m=+150.375770374" Jul 02 18:33:22.638904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:22.638870 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:22.639302 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:22.639160 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:22.643751 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:22.643727 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:23.111752 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:23.111723 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:33:23.156939 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:23.156904 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bd59f458-ld6fx"] Jul 02 18:33:28.971704 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:28.971665 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj"] Jul 02 18:33:28.974074 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:28.974056 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" Jul 02 18:33:28.976556 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:28.976529 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kcj48\"" Jul 02 18:33:28.976656 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:28.976556 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 02 18:33:28.976656 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:28.976556 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 02 18:33:28.976752 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:28.976539 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 02 18:33:28.976752 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:28.976556 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 02 18:33:28.984961 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:28.984937 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj"] Jul 02 18:33:29.068439 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.068401 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5d9m\" (UniqueName: \"kubernetes.io/projected/dfb9e7b8-1e97-4f54-aa23-5098784cc440-kube-api-access-j5d9m\") pod \"managed-serviceaccount-addon-agent-74b464c5fb-xtvtj\" (UID: \"dfb9e7b8-1e97-4f54-aa23-5098784cc440\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" Jul 02 18:33:29.068598 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.068444 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dfb9e7b8-1e97-4f54-aa23-5098784cc440-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74b464c5fb-xtvtj\" (UID: \"dfb9e7b8-1e97-4f54-aa23-5098784cc440\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" Jul 02 18:33:29.092187 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.092151 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch"] Jul 02 18:33:29.094609 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.094594 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.096489 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.096468 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 02 18:33:29.104035 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.104008 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch"] Jul 02 18:33:29.169529 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.169494 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlhz\" (UniqueName: \"kubernetes.io/projected/0fce8d58-00f6-44d4-9657-50a549bc6344-kube-api-access-ndlhz\") pod \"klusterlet-addon-workmgr-6fb488dcb5-2vkch\" (UID: \"0fce8d58-00f6-44d4-9657-50a549bc6344\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.169712 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.169543 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5d9m\" (UniqueName: \"kubernetes.io/projected/dfb9e7b8-1e97-4f54-aa23-5098784cc440-kube-api-access-j5d9m\") pod \"managed-serviceaccount-addon-agent-74b464c5fb-xtvtj\" (UID: \"dfb9e7b8-1e97-4f54-aa23-5098784cc440\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" Jul 02 18:33:29.169712 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.169602 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dfb9e7b8-1e97-4f54-aa23-5098784cc440-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74b464c5fb-xtvtj\" (UID: \"dfb9e7b8-1e97-4f54-aa23-5098784cc440\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" Jul 02 18:33:29.169712 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.169662 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fce8d58-00f6-44d4-9657-50a549bc6344-tmp\") pod \"klusterlet-addon-workmgr-6fb488dcb5-2vkch\" (UID: \"0fce8d58-00f6-44d4-9657-50a549bc6344\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.169877 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.169761 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0fce8d58-00f6-44d4-9657-50a549bc6344-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fb488dcb5-2vkch\" (UID: \"0fce8d58-00f6-44d4-9657-50a549bc6344\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.172113 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.172094 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dfb9e7b8-1e97-4f54-aa23-5098784cc440-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74b464c5fb-xtvtj\" (UID: \"dfb9e7b8-1e97-4f54-aa23-5098784cc440\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" Jul 02 18:33:29.177523 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.177502 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5d9m\" (UniqueName: \"kubernetes.io/projected/dfb9e7b8-1e97-4f54-aa23-5098784cc440-kube-api-access-j5d9m\") pod \"managed-serviceaccount-addon-agent-74b464c5fb-xtvtj\" (UID: \"dfb9e7b8-1e97-4f54-aa23-5098784cc440\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" Jul 02 18:33:29.270820 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.270731 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0fce8d58-00f6-44d4-9657-50a549bc6344-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fb488dcb5-2vkch\" (UID: \"0fce8d58-00f6-44d4-9657-50a549bc6344\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.270820 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.270774 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlhz\" (UniqueName: \"kubernetes.io/projected/0fce8d58-00f6-44d4-9657-50a549bc6344-kube-api-access-ndlhz\") pod \"klusterlet-addon-workmgr-6fb488dcb5-2vkch\" (UID: \"0fce8d58-00f6-44d4-9657-50a549bc6344\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.270820 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.270816 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fce8d58-00f6-44d4-9657-50a549bc6344-tmp\") pod \"klusterlet-addon-workmgr-6fb488dcb5-2vkch\" (UID: \"0fce8d58-00f6-44d4-9657-50a549bc6344\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.271250 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.271224 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fce8d58-00f6-44d4-9657-50a549bc6344-tmp\") pod \"klusterlet-addon-workmgr-6fb488dcb5-2vkch\" (UID: \"0fce8d58-00f6-44d4-9657-50a549bc6344\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.273278 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.273260 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0fce8d58-00f6-44d4-9657-50a549bc6344-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fb488dcb5-2vkch\" (UID: \"0fce8d58-00f6-44d4-9657-50a549bc6344\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.278809 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.278786 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlhz\" (UniqueName: \"kubernetes.io/projected/0fce8d58-00f6-44d4-9657-50a549bc6344-kube-api-access-ndlhz\") pod \"klusterlet-addon-workmgr-6fb488dcb5-2vkch\" (UID: \"0fce8d58-00f6-44d4-9657-50a549bc6344\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.299529 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.299511 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" Jul 02 18:33:29.404649 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.404606 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:29.425755 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.425729 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj"] Jul 02 18:33:29.428304 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:33:29.428277 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfb9e7b8_1e97_4f54_aa23_5098784cc440.slice/crio-9b0654d6b49387519c7752bc5e7b6dc50b7c92902147ee0cde19013ab405d6e6 WatchSource:0}: Error finding container 9b0654d6b49387519c7752bc5e7b6dc50b7c92902147ee0cde19013ab405d6e6: Status 404 returned error can't find the container with id 9b0654d6b49387519c7752bc5e7b6dc50b7c92902147ee0cde19013ab405d6e6 Jul 02 18:33:29.535003 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:29.534974 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch"] Jul 02 18:33:29.537296 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:33:29.537270 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fce8d58_00f6_44d4_9657_50a549bc6344.slice/crio-e6167c866188ce30290c6610eb0172e3a1f935e4291248222888856ddff0d2b0 WatchSource:0}: Error finding container e6167c866188ce30290c6610eb0172e3a1f935e4291248222888856ddff0d2b0: Status 404 returned error can't find the container with id e6167c866188ce30290c6610eb0172e3a1f935e4291248222888856ddff0d2b0 Jul 02 18:33:30.137171 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:30.137131 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" event={"ID":"0fce8d58-00f6-44d4-9657-50a549bc6344","Type":"ContainerStarted","Data":"e6167c866188ce30290c6610eb0172e3a1f935e4291248222888856ddff0d2b0"} Jul 02 18:33:30.138496 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:30.138467 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" event={"ID":"dfb9e7b8-1e97-4f54-aa23-5098784cc440","Type":"ContainerStarted","Data":"9b0654d6b49387519c7752bc5e7b6dc50b7c92902147ee0cde19013ab405d6e6"} Jul 02 18:33:34.158334 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:34.158288 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" event={"ID":"dfb9e7b8-1e97-4f54-aa23-5098784cc440","Type":"ContainerStarted","Data":"dd44b067d9fb3d1cb6928c5aa1b3cdbd16c91ff85db35bd4abb26bc8df8267f1"} Jul 02 18:33:34.159805 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:34.159778 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" event={"ID":"0fce8d58-00f6-44d4-9657-50a549bc6344","Type":"ContainerStarted","Data":"7826856b1d5587c176821286bc734a5a3722f8dba1e8ceeaca3fe5f24bf4d621"} Jul 02 18:33:34.159968 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:34.159953 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:34.161767 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:34.161749 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" Jul 02 18:33:34.172532 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:34.172489 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b464c5fb-xtvtj" podStartSLOduration=3.473511846 podStartE2EDuration="6.172475624s" totalImagesPullingTime="2.698963778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:33:34.171263325 +0000 UTC m=+171.457613167" watchObservedRunningTime="2026-07-02 18:33:34.172475624 +0000 UTC m=+171.458825467" Jul 02 18:33:34.185513 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:34.185469 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fb488dcb5-2vkch" podStartSLOduration=1.390612682 podStartE2EDuration="5.185456843s" totalImagesPullingTime="3.794844161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:33:34.184187543 +0000 UTC m=+171.470537388" watchObservedRunningTime="2026-07-02 18:33:34.185456843 +0000 UTC m=+171.471806686" Jul 02 18:33:48.182781 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.182716 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-55bd59f458-ld6fx" podUID="d2ee4054-7d43-4a4e-9805-859472549f78" containerName="console" containerID="cri-o://0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11" gracePeriod=15 Jul 02 18:33:48.421812 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.421787 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:33:48.541524 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.541427 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-service-ca\") pod \"d2ee4054-7d43-4a4e-9805-859472549f78\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " Jul 02 18:33:48.541524 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.541478 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-oauth-config\") pod \"d2ee4054-7d43-4a4e-9805-859472549f78\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " Jul 02 18:33:48.541524 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.541522 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-console-config\") pod \"d2ee4054-7d43-4a4e-9805-859472549f78\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " Jul 02 18:33:48.541845 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.541564 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-trusted-ca-bundle\") pod \"d2ee4054-7d43-4a4e-9805-859472549f78\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " Jul 02 18:33:48.541845 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.541601 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-oauth-serving-cert\") pod \"d2ee4054-7d43-4a4e-9805-859472549f78\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " Jul 02 18:33:48.541845 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.541630 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzfvd\" (UniqueName: \"kubernetes.io/projected/d2ee4054-7d43-4a4e-9805-859472549f78-kube-api-access-tzfvd\") pod \"d2ee4054-7d43-4a4e-9805-859472549f78\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " Jul 02 18:33:48.541845 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.541682 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-serving-cert\") pod \"d2ee4054-7d43-4a4e-9805-859472549f78\" (UID: \"d2ee4054-7d43-4a4e-9805-859472549f78\") " Jul 02 18:33:48.542065 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.542032 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-service-ca" (OuterVolumeSpecName: "service-ca") pod "d2ee4054-7d43-4a4e-9805-859472549f78" (UID: "d2ee4054-7d43-4a4e-9805-859472549f78"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:33:48.542119 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.542070 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d2ee4054-7d43-4a4e-9805-859472549f78" (UID: "d2ee4054-7d43-4a4e-9805-859472549f78"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:33:48.542119 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.542088 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-console-config" (OuterVolumeSpecName: "console-config") pod "d2ee4054-7d43-4a4e-9805-859472549f78" (UID: "d2ee4054-7d43-4a4e-9805-859472549f78"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:33:48.542197 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.542169 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d2ee4054-7d43-4a4e-9805-859472549f78" (UID: "d2ee4054-7d43-4a4e-9805-859472549f78"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:33:48.543889 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.543865 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d2ee4054-7d43-4a4e-9805-859472549f78" (UID: "d2ee4054-7d43-4a4e-9805-859472549f78"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:33:48.543999 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.543967 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ee4054-7d43-4a4e-9805-859472549f78-kube-api-access-tzfvd" (OuterVolumeSpecName: "kube-api-access-tzfvd") pod "d2ee4054-7d43-4a4e-9805-859472549f78" (UID: "d2ee4054-7d43-4a4e-9805-859472549f78"). InnerVolumeSpecName "kube-api-access-tzfvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:33:48.544042 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.543998 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d2ee4054-7d43-4a4e-9805-859472549f78" (UID: "d2ee4054-7d43-4a4e-9805-859472549f78"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:33:48.642704 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.642670 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-oauth-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:48.642704 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.642698 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfvd\" (UniqueName: \"kubernetes.io/projected/d2ee4054-7d43-4a4e-9805-859472549f78-kube-api-access-tzfvd\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:48.642704 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.642709 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:48.642704 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.642717 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-service-ca\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:48.642964 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.642727 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2ee4054-7d43-4a4e-9805-859472549f78-console-oauth-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:48.642964 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.642736 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-console-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:48.642964 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:48.642743 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2ee4054-7d43-4a4e-9805-859472549f78-trusted-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:33:49.218841 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:49.218803 2552 generic.go:356] "Generic (PLEG): container finished" podID="d2ee4054-7d43-4a4e-9805-859472549f78" containerID="0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11" exitCode=0 Jul 02 18:33:49.218841 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:49.218847 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd59f458-ld6fx" event={"ID":"d2ee4054-7d43-4a4e-9805-859472549f78","Type":"ContainerDied","Data":"0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11"} Jul 02 18:33:49.219392 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:49.218864 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd59f458-ld6fx" Jul 02 18:33:49.219392 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:49.218913 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd59f458-ld6fx" event={"ID":"d2ee4054-7d43-4a4e-9805-859472549f78","Type":"ContainerDied","Data":"1d624ad738302019fc0a2a359a336c7a164e82f951ccf6719a8943cf1bf3165c"} Jul 02 18:33:49.219392 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:49.218935 2552 scope.go:117] "RemoveContainer" containerID="0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11" Jul 02 18:33:49.227489 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:49.227470 2552 scope.go:117] "RemoveContainer" containerID="0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11" Jul 02 18:33:49.227780 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:33:49.227758 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11\": container with ID starting with 0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11 not found: ID does not exist" containerID="0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11" Jul 02 18:33:49.227857 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:49.227791 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11"} err="failed to get container status \"0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11\": rpc error: code = NotFound desc = could not find container \"0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11\": container with ID starting with 0856a3d0150b5d24cf05b2f628efba32c50364e34418fa084e782f6abd750b11 not found: ID does not exist" Jul 02 18:33:49.238372 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:49.238338 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bd59f458-ld6fx"] Jul 02 18:33:49.239936 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:49.239915 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55bd59f458-ld6fx"] Jul 02 18:33:51.236019 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:33:51.235985 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ee4054-7d43-4a4e-9805-859472549f78" path="/var/lib/kubelet/pods/d2ee4054-7d43-4a4e-9805-859472549f78/volumes" Jul 02 18:34:40.257063 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.257030 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf"] Jul 02 18:34:40.257504 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.257340 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ee4054-7d43-4a4e-9805-859472549f78" containerName="console" Jul 02 18:34:40.257504 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.257350 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ee4054-7d43-4a4e-9805-859472549f78" containerName="console" Jul 02 18:34:40.257504 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.257401 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ee4054-7d43-4a4e-9805-859472549f78" containerName="console" Jul 02 18:34:40.259526 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.259508 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.261571 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.261554 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 02 18:34:40.261661 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.261602 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 02 18:34:40.262245 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.262221 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sv8zm\"" Jul 02 18:34:40.269015 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.268980 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf"] Jul 02 18:34:40.388624 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.388593 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.388817 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.388669 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.388817 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.388729 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbxl\" (UniqueName: \"kubernetes.io/projected/2774632b-da31-4864-9926-efffdcb13b23-kube-api-access-hsbxl\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.490058 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.490029 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.490207 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.490080 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbxl\" (UniqueName: \"kubernetes.io/projected/2774632b-da31-4864-9926-efffdcb13b23-kube-api-access-hsbxl\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.490207 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.490136 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.490404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.490384 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.490442 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.490418 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.497760 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.497739 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbxl\" (UniqueName: \"kubernetes.io/projected/2774632b-da31-4864-9926-efffdcb13b23-kube-api-access-hsbxl\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.572719 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.572622 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:40.695975 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:40.695951 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf"] Jul 02 18:34:40.698249 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:34:40.698224 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2774632b_da31_4864_9926_efffdcb13b23.slice/crio-f6df977f20970ff6aee52fb4f2fd86e48f62c746704e5a3cfcc9cb0e8c381716 WatchSource:0}: Error finding container f6df977f20970ff6aee52fb4f2fd86e48f62c746704e5a3cfcc9cb0e8c381716: Status 404 returned error can't find the container with id f6df977f20970ff6aee52fb4f2fd86e48f62c746704e5a3cfcc9cb0e8c381716 Jul 02 18:34:41.417370 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:41.417331 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" event={"ID":"2774632b-da31-4864-9926-efffdcb13b23","Type":"ContainerStarted","Data":"f6df977f20970ff6aee52fb4f2fd86e48f62c746704e5a3cfcc9cb0e8c381716"} Jul 02 18:34:46.440609 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:46.440568 2552 generic.go:356] "Generic (PLEG): container finished" podID="2774632b-da31-4864-9926-efffdcb13b23" containerID="8104df18592d93d25938be6d21d17b7d39c0cad9347445e67925753dbd794176" exitCode=0 Jul 02 18:34:46.441002 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:46.440619 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" event={"ID":"2774632b-da31-4864-9926-efffdcb13b23","Type":"ContainerDied","Data":"8104df18592d93d25938be6d21d17b7d39c0cad9347445e67925753dbd794176"} Jul 02 18:34:49.455539 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:49.455503 2552 generic.go:356] "Generic (PLEG): container finished" podID="2774632b-da31-4864-9926-efffdcb13b23" containerID="5a202f3034574358088e4de29db003ccb5753fe37c2017544c7afa29f9441fe2" exitCode=0 Jul 02 18:34:49.455942 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:49.455595 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" event={"ID":"2774632b-da31-4864-9926-efffdcb13b23","Type":"ContainerDied","Data":"5a202f3034574358088e4de29db003ccb5753fe37c2017544c7afa29f9441fe2"} Jul 02 18:34:57.491834 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:57.491795 2552 generic.go:356] "Generic (PLEG): container finished" podID="2774632b-da31-4864-9926-efffdcb13b23" containerID="bf766163c3a8fd7156d5d68a4417cb00172511206d1d7cbe3bba9d1a41655432" exitCode=0 Jul 02 18:34:57.492201 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:57.491842 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" event={"ID":"2774632b-da31-4864-9926-efffdcb13b23","Type":"ContainerDied","Data":"bf766163c3a8fd7156d5d68a4417cb00172511206d1d7cbe3bba9d1a41655432"} Jul 02 18:34:58.619245 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.619218 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:58.648647 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.648596 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-bundle\") pod \"2774632b-da31-4864-9926-efffdcb13b23\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " Jul 02 18:34:58.648801 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.648709 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-util\") pod \"2774632b-da31-4864-9926-efffdcb13b23\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " Jul 02 18:34:58.648801 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.648755 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsbxl\" (UniqueName: \"kubernetes.io/projected/2774632b-da31-4864-9926-efffdcb13b23-kube-api-access-hsbxl\") pod \"2774632b-da31-4864-9926-efffdcb13b23\" (UID: \"2774632b-da31-4864-9926-efffdcb13b23\") " Jul 02 18:34:58.649280 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.649250 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-bundle" (OuterVolumeSpecName: "bundle") pod "2774632b-da31-4864-9926-efffdcb13b23" (UID: "2774632b-da31-4864-9926-efffdcb13b23"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 18:34:58.651003 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.650971 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2774632b-da31-4864-9926-efffdcb13b23-kube-api-access-hsbxl" (OuterVolumeSpecName: "kube-api-access-hsbxl") pod "2774632b-da31-4864-9926-efffdcb13b23" (UID: "2774632b-da31-4864-9926-efffdcb13b23"). InnerVolumeSpecName "kube-api-access-hsbxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:34:58.653310 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.653288 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-util" (OuterVolumeSpecName: "util") pod "2774632b-da31-4864-9926-efffdcb13b23" (UID: "2774632b-da31-4864-9926-efffdcb13b23"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 18:34:58.750254 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.750159 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:34:58.750254 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.750189 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2774632b-da31-4864-9926-efffdcb13b23-util\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:34:58.750254 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:58.750201 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsbxl\" (UniqueName: \"kubernetes.io/projected/2774632b-da31-4864-9926-efffdcb13b23-kube-api-access-hsbxl\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:34:59.500850 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:59.500754 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" Jul 02 18:34:59.500850 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:59.500754 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5896qpgf" event={"ID":"2774632b-da31-4864-9926-efffdcb13b23","Type":"ContainerDied","Data":"f6df977f20970ff6aee52fb4f2fd86e48f62c746704e5a3cfcc9cb0e8c381716"} Jul 02 18:34:59.501013 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:34:59.500871 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6df977f20970ff6aee52fb4f2fd86e48f62c746704e5a3cfcc9cb0e8c381716" Jul 02 18:35:02.347509 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.347478 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86"] Jul 02 18:35:02.347934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.347844 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2774632b-da31-4864-9926-efffdcb13b23" containerName="util" Jul 02 18:35:02.347934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.347856 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2774632b-da31-4864-9926-efffdcb13b23" containerName="util" Jul 02 18:35:02.347934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.347868 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2774632b-da31-4864-9926-efffdcb13b23" containerName="pull" Jul 02 18:35:02.347934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.347873 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2774632b-da31-4864-9926-efffdcb13b23" containerName="pull" Jul 02 18:35:02.347934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.347885 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2774632b-da31-4864-9926-efffdcb13b23" containerName="extract" Jul 02 18:35:02.347934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.347891 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2774632b-da31-4864-9926-efffdcb13b23" containerName="extract" Jul 02 18:35:02.347934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.347937 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2774632b-da31-4864-9926-efffdcb13b23" containerName="extract" Jul 02 18:35:02.356797 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.356774 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:02.358804 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.358781 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 02 18:35:02.359019 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.359000 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 02 18:35:02.359096 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.359021 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 02 18:35:02.359096 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.359025 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-fcrbk\"" Jul 02 18:35:02.362373 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.362352 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86"] Jul 02 18:35:02.383097 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.383072 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8vp\" (UniqueName: \"kubernetes.io/projected/c17edba2-8268-4354-9013-6035b0635e62-kube-api-access-sp8vp\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xfj86\" (UID: \"c17edba2-8268-4354-9013-6035b0635e62\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:02.383215 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.383133 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c17edba2-8268-4354-9013-6035b0635e62-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xfj86\" (UID: \"c17edba2-8268-4354-9013-6035b0635e62\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:02.483754 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.483722 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8vp\" (UniqueName: \"kubernetes.io/projected/c17edba2-8268-4354-9013-6035b0635e62-kube-api-access-sp8vp\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xfj86\" (UID: \"c17edba2-8268-4354-9013-6035b0635e62\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:02.483916 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.483813 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c17edba2-8268-4354-9013-6035b0635e62-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xfj86\" (UID: \"c17edba2-8268-4354-9013-6035b0635e62\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:02.486096 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.486072 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c17edba2-8268-4354-9013-6035b0635e62-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xfj86\" (UID: \"c17edba2-8268-4354-9013-6035b0635e62\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:02.492818 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.492786 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8vp\" (UniqueName: \"kubernetes.io/projected/c17edba2-8268-4354-9013-6035b0635e62-kube-api-access-sp8vp\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xfj86\" (UID: \"c17edba2-8268-4354-9013-6035b0635e62\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:02.667345 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.667257 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:02.831889 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:02.831865 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86"] Jul 02 18:35:02.834442 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:35:02.834410 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17edba2_8268_4354_9013_6035b0635e62.slice/crio-de2733a4a48f01d862b8228960eb6ae45ab51c9f70153eec78a549777d446c37 WatchSource:0}: Error finding container de2733a4a48f01d862b8228960eb6ae45ab51c9f70153eec78a549777d446c37: Status 404 returned error can't find the container with id de2733a4a48f01d862b8228960eb6ae45ab51c9f70153eec78a549777d446c37 Jul 02 18:35:03.517233 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:03.517198 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" event={"ID":"c17edba2-8268-4354-9013-6035b0635e62","Type":"ContainerStarted","Data":"de2733a4a48f01d862b8228960eb6ae45ab51c9f70153eec78a549777d446c37"} Jul 02 18:35:07.539881 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:07.539846 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" event={"ID":"c17edba2-8268-4354-9013-6035b0635e62","Type":"ContainerStarted","Data":"f3346d845d3e6d3250d380a1ebe84281c776ed5ac2513d79fcaf77d52e87f4e6"} Jul 02 18:35:07.540322 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:07.539991 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:07.560470 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:07.560416 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" podStartSLOduration=1.386245537 podStartE2EDuration="5.560399829s" totalImagesPullingTime="4.174154292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:35:07.558332584 +0000 UTC m=+264.844682426" watchObservedRunningTime="2026-07-02 18:35:07.560399829 +0000 UTC m=+264.846749672" Jul 02 18:35:08.037774 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.037733 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-drk98"] Jul 02 18:35:08.053717 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.053682 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-drk98"] Jul 02 18:35:08.053875 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.053791 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.056181 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.056156 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Jul 02 18:35:08.056334 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.056283 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 02 18:35:08.056408 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.056329 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7x2cc\"" Jul 02 18:35:08.131854 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.131814 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/770e9fa4-af79-4e97-b48a-c333d858fe88-certificates\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.132032 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.131878 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl9cb\" (UniqueName: \"kubernetes.io/projected/770e9fa4-af79-4e97-b48a-c333d858fe88-kube-api-access-hl9cb\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.132032 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.131939 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/770e9fa4-af79-4e97-b48a-c333d858fe88-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.233186 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.233152 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl9cb\" (UniqueName: \"kubernetes.io/projected/770e9fa4-af79-4e97-b48a-c333d858fe88-kube-api-access-hl9cb\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.233186 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.233187 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/770e9fa4-af79-4e97-b48a-c333d858fe88-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.233400 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.233325 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/770e9fa4-af79-4e97-b48a-c333d858fe88-certificates\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.233500 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:35:08.233482 2552 secret.go:281] references non-existent secret key: tls.crt Jul 02 18:35:08.233571 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.233497 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/770e9fa4-af79-4e97-b48a-c333d858fe88-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.233571 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:35:08.233507 2552 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Jul 02 18:35:08.233571 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:35:08.233531 2552 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found Jul 02 18:35:08.233571 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:35:08.233557 2552 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-drk98: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found] Jul 02 18:35:08.233769 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:35:08.233626 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/770e9fa4-af79-4e97-b48a-c333d858fe88-certificates podName:770e9fa4-af79-4e97-b48a-c333d858fe88 nodeName:}" failed. No retries permitted until 2026-07-02 18:35:08.733606117 +0000 UTC m=+266.019955941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/770e9fa4-af79-4e97-b48a-c333d858fe88-certificates") pod "keda-metrics-apiserver-559f75f947-drk98" (UID: "770e9fa4-af79-4e97-b48a-c333d858fe88") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found] Jul 02 18:35:08.241719 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.241695 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl9cb\" (UniqueName: \"kubernetes.io/projected/770e9fa4-af79-4e97-b48a-c333d858fe88-kube-api-access-hl9cb\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.620733 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.620696 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-c7pjl"] Jul 02 18:35:08.641910 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.641883 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-c7pjl"] Jul 02 18:35:08.642054 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.641976 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:08.644112 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.644093 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 02 18:35:08.737095 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.737052 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62ph5\" (UniqueName: \"kubernetes.io/projected/db551038-33af-4c70-b057-089c94cf490b-kube-api-access-62ph5\") pod \"keda-admission-5fbd5c4644-c7pjl\" (UID: \"db551038-33af-4c70-b057-089c94cf490b\") " pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:08.737295 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.737126 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/770e9fa4-af79-4e97-b48a-c333d858fe88-certificates\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.737295 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.737145 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db551038-33af-4c70-b057-089c94cf490b-certificates\") pod \"keda-admission-5fbd5c4644-c7pjl\" (UID: \"db551038-33af-4c70-b057-089c94cf490b\") " pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:08.739791 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.739762 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/770e9fa4-af79-4e97-b48a-c333d858fe88-certificates\") pod \"keda-metrics-apiserver-559f75f947-drk98\" (UID: \"770e9fa4-af79-4e97-b48a-c333d858fe88\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:08.837861 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.837825 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62ph5\" (UniqueName: \"kubernetes.io/projected/db551038-33af-4c70-b057-089c94cf490b-kube-api-access-62ph5\") pod \"keda-admission-5fbd5c4644-c7pjl\" (UID: \"db551038-33af-4c70-b057-089c94cf490b\") " pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:08.838030 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.837896 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db551038-33af-4c70-b057-089c94cf490b-certificates\") pod \"keda-admission-5fbd5c4644-c7pjl\" (UID: \"db551038-33af-4c70-b057-089c94cf490b\") " pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:08.838030 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:35:08.838000 2552 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Jul 02 18:35:08.838030 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:35:08.838019 2552 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-c7pjl: secret "keda-admission-webhooks-certs" not found Jul 02 18:35:08.838125 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:35:08.838073 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db551038-33af-4c70-b057-089c94cf490b-certificates podName:db551038-33af-4c70-b057-089c94cf490b nodeName:}" failed. No retries permitted until 2026-07-02 18:35:09.338057168 +0000 UTC m=+266.624406988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/db551038-33af-4c70-b057-089c94cf490b-certificates") pod "keda-admission-5fbd5c4644-c7pjl" (UID: "db551038-33af-4c70-b057-089c94cf490b") : secret "keda-admission-webhooks-certs" not found Jul 02 18:35:08.856012 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.855982 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62ph5\" (UniqueName: \"kubernetes.io/projected/db551038-33af-4c70-b057-089c94cf490b-kube-api-access-62ph5\") pod \"keda-admission-5fbd5c4644-c7pjl\" (UID: \"db551038-33af-4c70-b057-089c94cf490b\") " pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:08.966070 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:08.966032 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:09.130579 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:09.130542 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-drk98"] Jul 02 18:35:09.134050 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:35:09.134022 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770e9fa4_af79_4e97_b48a_c333d858fe88.slice/crio-b08e4d78466671133b150d20bf43a44ef99b2c713847b1319e55218692385b4d WatchSource:0}: Error finding container b08e4d78466671133b150d20bf43a44ef99b2c713847b1319e55218692385b4d: Status 404 returned error can't find the container with id b08e4d78466671133b150d20bf43a44ef99b2c713847b1319e55218692385b4d Jul 02 18:35:09.342548 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:09.342452 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db551038-33af-4c70-b057-089c94cf490b-certificates\") pod \"keda-admission-5fbd5c4644-c7pjl\" (UID: \"db551038-33af-4c70-b057-089c94cf490b\") " pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:09.345040 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:09.345014 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db551038-33af-4c70-b057-089c94cf490b-certificates\") pod \"keda-admission-5fbd5c4644-c7pjl\" (UID: \"db551038-33af-4c70-b057-089c94cf490b\") " pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:09.550711 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:09.550673 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" event={"ID":"770e9fa4-af79-4e97-b48a-c333d858fe88","Type":"ContainerStarted","Data":"b08e4d78466671133b150d20bf43a44ef99b2c713847b1319e55218692385b4d"} Jul 02 18:35:09.551786 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:09.551774 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:09.708222 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:09.708162 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-c7pjl"] Jul 02 18:35:09.710787 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:35:09.710757 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb551038_33af_4c70_b057_089c94cf490b.slice/crio-e9eb0ad35337510ed6a6a13733187f11ab01506d6d8742cd3a3b03ed9f693cae WatchSource:0}: Error finding container e9eb0ad35337510ed6a6a13733187f11ab01506d6d8742cd3a3b03ed9f693cae: Status 404 returned error can't find the container with id e9eb0ad35337510ed6a6a13733187f11ab01506d6d8742cd3a3b03ed9f693cae Jul 02 18:35:10.558879 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:10.558839 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" event={"ID":"db551038-33af-4c70-b057-089c94cf490b","Type":"ContainerStarted","Data":"e9eb0ad35337510ed6a6a13733187f11ab01506d6d8742cd3a3b03ed9f693cae"} Jul 02 18:35:11.565045 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:11.565009 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" event={"ID":"db551038-33af-4c70-b057-089c94cf490b","Type":"ContainerStarted","Data":"982eece354be761e2501fd35325ff822b9c66230b80e615647b6c305a707e6b8"} Jul 02 18:35:11.565399 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:11.565131 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:11.580279 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:11.580229 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" podStartSLOduration=1.8814861139999999 podStartE2EDuration="3.580211828s" totalImagesPullingTime="1.698725714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:35:11.579371217 +0000 UTC m=+268.865721058" watchObservedRunningTime="2026-07-02 18:35:11.580211828 +0000 UTC m=+268.866561674" Jul 02 18:35:22.615198 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:22.615146 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" event={"ID":"770e9fa4-af79-4e97-b48a-c333d858fe88","Type":"ContainerStarted","Data":"929cc70955a8633ca610e9676ca2983c993dd245d4e193d9e8b1d2ecf2a5a78e"} Jul 02 18:35:22.615560 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:22.615344 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:22.632078 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:22.632030 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" podStartSLOduration=1.286532395 podStartE2EDuration="14.632013507s" totalImagesPullingTime="13.345481112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:35:22.629791818 +0000 UTC m=+279.916141670" watchObservedRunningTime="2026-07-02 18:35:22.632013507 +0000 UTC m=+279.918363350" Jul 02 18:35:28.547525 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:28.547495 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xfj86" Jul 02 18:35:32.573371 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:32.573337 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-c7pjl" Jul 02 18:35:33.624140 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:33.624051 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-drk98" Jul 02 18:35:43.097464 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:35:43.097436 2552 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 18:36:13.817177 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.817145 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7cd6c584bf-2glzm"] Jul 02 18:36:13.819479 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.819460 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:13.821362 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.821342 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 02 18:36:13.821764 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.821742 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 02 18:36:13.821865 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.821742 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-m5znp\"" Jul 02 18:36:13.822490 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.822474 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 02 18:36:13.830875 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.830855 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd6c584bf-2glzm"] Jul 02 18:36:13.895870 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.895836 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43403af9-37bd-4389-b43d-91639d5c6210-cert\") pod \"kserve-controller-manager-7cd6c584bf-2glzm\" (UID: \"43403af9-37bd-4389-b43d-91639d5c6210\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:13.896041 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.895919 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/43403af9-37bd-4389-b43d-91639d5c6210-kube-api-access-h9jgk\") pod \"kserve-controller-manager-7cd6c584bf-2glzm\" (UID: \"43403af9-37bd-4389-b43d-91639d5c6210\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:13.997012 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.996964 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/43403af9-37bd-4389-b43d-91639d5c6210-kube-api-access-h9jgk\") pod \"kserve-controller-manager-7cd6c584bf-2glzm\" (UID: \"43403af9-37bd-4389-b43d-91639d5c6210\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:13.997203 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.997028 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43403af9-37bd-4389-b43d-91639d5c6210-cert\") pod \"kserve-controller-manager-7cd6c584bf-2glzm\" (UID: \"43403af9-37bd-4389-b43d-91639d5c6210\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:13.999363 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:13.999333 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43403af9-37bd-4389-b43d-91639d5c6210-cert\") pod \"kserve-controller-manager-7cd6c584bf-2glzm\" (UID: \"43403af9-37bd-4389-b43d-91639d5c6210\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:14.006108 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:14.006088 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/43403af9-37bd-4389-b43d-91639d5c6210-kube-api-access-h9jgk\") pod \"kserve-controller-manager-7cd6c584bf-2glzm\" (UID: \"43403af9-37bd-4389-b43d-91639d5c6210\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:14.131312 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:14.131217 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:14.261428 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:14.261353 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd6c584bf-2glzm"] Jul 02 18:36:14.264614 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:36:14.264586 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43403af9_37bd_4389_b43d_91639d5c6210.slice/crio-6f6de55d00f83f495edc88a66fbf67791b8380d989b979542fe90a740331fcef WatchSource:0}: Error finding container 6f6de55d00f83f495edc88a66fbf67791b8380d989b979542fe90a740331fcef: Status 404 returned error can't find the container with id 6f6de55d00f83f495edc88a66fbf67791b8380d989b979542fe90a740331fcef Jul 02 18:36:14.266347 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:14.266329 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 18:36:14.851948 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:14.851912 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" event={"ID":"43403af9-37bd-4389-b43d-91639d5c6210","Type":"ContainerStarted","Data":"6f6de55d00f83f495edc88a66fbf67791b8380d989b979542fe90a740331fcef"} Jul 02 18:36:17.868091 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:17.868051 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" event={"ID":"43403af9-37bd-4389-b43d-91639d5c6210","Type":"ContainerStarted","Data":"76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e"} Jul 02 18:36:17.868525 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:17.868180 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:17.882344 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:17.882297 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" podStartSLOduration=2.142875321 podStartE2EDuration="4.8822831s" totalImagesPullingTime="2.739407779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:36:17.88203562 +0000 UTC m=+335.168385462" watchObservedRunningTime="2026-07-02 18:36:17.8822831 +0000 UTC m=+335.168632944" Jul 02 18:36:48.877426 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:48.877352 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:49.409750 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.409716 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7cd6c584bf-2glzm"] Jul 02 18:36:49.409929 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.409893 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" podUID="43403af9-37bd-4389-b43d-91639d5c6210" containerName="manager" containerID="cri-o://76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e" gracePeriod=10 Jul 02 18:36:49.433191 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.433167 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7cd6c584bf-2xwxb"] Jul 02 18:36:49.436690 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.436675 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:36:49.443495 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.443470 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd6c584bf-2xwxb"] Jul 02 18:36:49.497705 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.497680 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vkd\" (UniqueName: \"kubernetes.io/projected/d94d3095-e3e6-472f-b7ea-87ef50a87b57-kube-api-access-d7vkd\") pod \"kserve-controller-manager-7cd6c584bf-2xwxb\" (UID: \"d94d3095-e3e6-472f-b7ea-87ef50a87b57\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:36:49.497886 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.497727 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d94d3095-e3e6-472f-b7ea-87ef50a87b57-cert\") pod \"kserve-controller-manager-7cd6c584bf-2xwxb\" (UID: \"d94d3095-e3e6-472f-b7ea-87ef50a87b57\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:36:49.598302 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.598276 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vkd\" (UniqueName: \"kubernetes.io/projected/d94d3095-e3e6-472f-b7ea-87ef50a87b57-kube-api-access-d7vkd\") pod \"kserve-controller-manager-7cd6c584bf-2xwxb\" (UID: \"d94d3095-e3e6-472f-b7ea-87ef50a87b57\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:36:49.598451 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.598324 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d94d3095-e3e6-472f-b7ea-87ef50a87b57-cert\") pod \"kserve-controller-manager-7cd6c584bf-2xwxb\" (UID: \"d94d3095-e3e6-472f-b7ea-87ef50a87b57\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:36:49.600543 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.600521 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d94d3095-e3e6-472f-b7ea-87ef50a87b57-cert\") pod \"kserve-controller-manager-7cd6c584bf-2xwxb\" (UID: \"d94d3095-e3e6-472f-b7ea-87ef50a87b57\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:36:49.606252 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.606230 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vkd\" (UniqueName: \"kubernetes.io/projected/d94d3095-e3e6-472f-b7ea-87ef50a87b57-kube-api-access-d7vkd\") pod \"kserve-controller-manager-7cd6c584bf-2xwxb\" (UID: \"d94d3095-e3e6-472f-b7ea-87ef50a87b57\") " pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:36:49.650984 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.650961 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:49.698894 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.698866 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/43403af9-37bd-4389-b43d-91639d5c6210-kube-api-access-h9jgk\") pod \"43403af9-37bd-4389-b43d-91639d5c6210\" (UID: \"43403af9-37bd-4389-b43d-91639d5c6210\") " Jul 02 18:36:49.699053 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.698910 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43403af9-37bd-4389-b43d-91639d5c6210-cert\") pod \"43403af9-37bd-4389-b43d-91639d5c6210\" (UID: \"43403af9-37bd-4389-b43d-91639d5c6210\") " Jul 02 18:36:49.701056 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.701026 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43403af9-37bd-4389-b43d-91639d5c6210-cert" (OuterVolumeSpecName: "cert") pod "43403af9-37bd-4389-b43d-91639d5c6210" (UID: "43403af9-37bd-4389-b43d-91639d5c6210"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:36:49.701153 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.701084 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43403af9-37bd-4389-b43d-91639d5c6210-kube-api-access-h9jgk" (OuterVolumeSpecName: "kube-api-access-h9jgk") pod "43403af9-37bd-4389-b43d-91639d5c6210" (UID: "43403af9-37bd-4389-b43d-91639d5c6210"). InnerVolumeSpecName "kube-api-access-h9jgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:36:49.799762 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.799731 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/43403af9-37bd-4389-b43d-91639d5c6210-kube-api-access-h9jgk\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:36:49.799762 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.799757 2552 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43403af9-37bd-4389-b43d-91639d5c6210-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:36:49.803578 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.803556 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:36:49.932945 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:49.932908 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd6c584bf-2xwxb"] Jul 02 18:36:49.935737 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:36:49.935704 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94d3095_e3e6_472f_b7ea_87ef50a87b57.slice/crio-98e7202e75efdce28e08c28c01e24c97c34fe7c0a5bbbbfc070b7641decfa4c9 WatchSource:0}: Error finding container 98e7202e75efdce28e08c28c01e24c97c34fe7c0a5bbbbfc070b7641decfa4c9: Status 404 returned error can't find the container with id 98e7202e75efdce28e08c28c01e24c97c34fe7c0a5bbbbfc070b7641decfa4c9 Jul 02 18:36:50.008002 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.007907 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" event={"ID":"d94d3095-e3e6-472f-b7ea-87ef50a87b57","Type":"ContainerStarted","Data":"98e7202e75efdce28e08c28c01e24c97c34fe7c0a5bbbbfc070b7641decfa4c9"} Jul 02 18:36:50.009110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.009080 2552 generic.go:356] "Generic (PLEG): container finished" podID="43403af9-37bd-4389-b43d-91639d5c6210" containerID="76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e" exitCode=0 Jul 02 18:36:50.009230 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.009119 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" event={"ID":"43403af9-37bd-4389-b43d-91639d5c6210","Type":"ContainerDied","Data":"76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e"} Jul 02 18:36:50.009230 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.009127 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" Jul 02 18:36:50.009230 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.009151 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd6c584bf-2glzm" event={"ID":"43403af9-37bd-4389-b43d-91639d5c6210","Type":"ContainerDied","Data":"6f6de55d00f83f495edc88a66fbf67791b8380d989b979542fe90a740331fcef"} Jul 02 18:36:50.009230 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.009174 2552 scope.go:117] "RemoveContainer" containerID="76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e" Jul 02 18:36:50.017581 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.017563 2552 scope.go:117] "RemoveContainer" containerID="76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e" Jul 02 18:36:50.017856 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:36:50.017838 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e\": container with ID starting with 76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e not found: ID does not exist" containerID="76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e" Jul 02 18:36:50.017927 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.017864 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e"} err="failed to get container status \"76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e\": rpc error: code = NotFound desc = could not find container \"76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e\": container with ID starting with 76ce9f83d0acbea91835ccccd348322837284b7695ab8f7d119851e726586e8e not found: ID does not exist" Jul 02 18:36:50.030488 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.030467 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7cd6c584bf-2glzm"] Jul 02 18:36:50.036702 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:50.036683 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7cd6c584bf-2glzm"] Jul 02 18:36:51.014458 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:51.014421 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" event={"ID":"d94d3095-e3e6-472f-b7ea-87ef50a87b57","Type":"ContainerStarted","Data":"bd68ed9b6da151176a23a58d2e7fa573b63caae23108c638a39b62b60d42d2db"} Jul 02 18:36:51.014950 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:51.014483 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:36:51.030550 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:51.030508 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" podStartSLOduration=1.691800032 podStartE2EDuration="2.030492184s" totalImagesPullingTime="338.692152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:36:51.028595924 +0000 UTC m=+368.314945767" watchObservedRunningTime="2026-07-02 18:36:51.030492184 +0000 UTC m=+368.316842027" Jul 02 18:36:51.241049 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:36:51.237823 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43403af9-37bd-4389-b43d-91639d5c6210" path="/var/lib/kubelet/pods/43403af9-37bd-4389-b43d-91639d5c6210/volumes" Jul 02 18:37:12.886888 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:12.886856 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b858d7ddb-qmprg"] Jul 02 18:37:12.887357 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:12.887338 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43403af9-37bd-4389-b43d-91639d5c6210" containerName="manager" Jul 02 18:37:12.887428 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:12.887361 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="43403af9-37bd-4389-b43d-91639d5c6210" containerName="manager" Jul 02 18:37:12.887486 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:12.887475 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="43403af9-37bd-4389-b43d-91639d5c6210" containerName="manager" Jul 02 18:37:12.890484 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:12.890464 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:12.901918 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:12.901896 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b858d7ddb-qmprg"] Jul 02 18:37:13.005063 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.005027 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95ba0dee-2108-4859-a2c5-ab6ed665840b-console-serving-cert\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.005063 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.005063 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccg8t\" (UniqueName: \"kubernetes.io/projected/95ba0dee-2108-4859-a2c5-ab6ed665840b-kube-api-access-ccg8t\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.005278 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.005102 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95ba0dee-2108-4859-a2c5-ab6ed665840b-console-oauth-config\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.005278 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.005184 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-console-config\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.005278 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.005214 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-trusted-ca-bundle\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.005278 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.005235 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-service-ca\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.005278 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.005260 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-oauth-serving-cert\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.105725 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.105689 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95ba0dee-2108-4859-a2c5-ab6ed665840b-console-oauth-config\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.105895 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.105778 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-console-config\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.105895 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.105807 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-trusted-ca-bundle\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.105895 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.105834 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-service-ca\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.105895 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.105857 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-oauth-serving-cert\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.105895 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.105888 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95ba0dee-2108-4859-a2c5-ab6ed665840b-console-serving-cert\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.106103 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.105914 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccg8t\" (UniqueName: \"kubernetes.io/projected/95ba0dee-2108-4859-a2c5-ab6ed665840b-kube-api-access-ccg8t\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.107059 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.107020 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-service-ca\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.107192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.107067 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-trusted-ca-bundle\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.107192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.107089 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-console-config\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.107192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.107109 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95ba0dee-2108-4859-a2c5-ab6ed665840b-oauth-serving-cert\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.108458 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.108433 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95ba0dee-2108-4859-a2c5-ab6ed665840b-console-serving-cert\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.108688 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.108667 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95ba0dee-2108-4859-a2c5-ab6ed665840b-console-oauth-config\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.113295 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.113269 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccg8t\" (UniqueName: \"kubernetes.io/projected/95ba0dee-2108-4859-a2c5-ab6ed665840b-kube-api-access-ccg8t\") pod \"console-6b858d7ddb-qmprg\" (UID: \"95ba0dee-2108-4859-a2c5-ab6ed665840b\") " pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.200129 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.200102 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:13.329009 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:13.328989 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b858d7ddb-qmprg"] Jul 02 18:37:13.331587 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:37:13.331560 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ba0dee_2108_4859_a2c5_ab6ed665840b.slice/crio-7f2cb368d3db18d21e11bf93e8718a941395962e908fa6c3713c37970653c193 WatchSource:0}: Error finding container 7f2cb368d3db18d21e11bf93e8718a941395962e908fa6c3713c37970653c193: Status 404 returned error can't find the container with id 7f2cb368d3db18d21e11bf93e8718a941395962e908fa6c3713c37970653c193 Jul 02 18:37:14.118175 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:14.118137 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b858d7ddb-qmprg" event={"ID":"95ba0dee-2108-4859-a2c5-ab6ed665840b","Type":"ContainerStarted","Data":"f071f269072388a6aa1f75e262a4d48e56f56ac95648539ac1a10372452ac7e3"} Jul 02 18:37:14.118498 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:14.118230 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b858d7ddb-qmprg" event={"ID":"95ba0dee-2108-4859-a2c5-ab6ed665840b","Type":"ContainerStarted","Data":"7f2cb368d3db18d21e11bf93e8718a941395962e908fa6c3713c37970653c193"} Jul 02 18:37:14.133088 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:14.133031 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b858d7ddb-qmprg" podStartSLOduration=2.133016255 podStartE2EDuration="2.133016255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:37:14.132776964 +0000 UTC m=+391.419126807" watchObservedRunningTime="2026-07-02 18:37:14.133016255 +0000 UTC m=+391.419366103" Jul 02 18:37:22.025465 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:22.025436 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7cd6c584bf-2xwxb" Jul 02 18:37:23.171442 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.171406 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-l4dmh"] Jul 02 18:37:23.175487 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.175468 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:23.177270 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.177254 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jul 02 18:37:23.177510 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.177493 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-w77z6\"" Jul 02 18:37:23.184586 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.184563 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-l4dmh"] Jul 02 18:37:23.190944 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.190923 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-nvr8r"] Jul 02 18:37:23.194059 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.194044 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:23.196528 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.196510 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-62nmd\"" Jul 02 18:37:23.196528 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.196524 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 02 18:37:23.200262 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.200243 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:23.200370 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.200295 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:23.203545 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.203525 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-nvr8r"] Jul 02 18:37:23.205702 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.205683 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:23.285592 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.285570 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvnj\" (UniqueName: \"kubernetes.io/projected/8ed463a4-8399-44b5-b2e1-8d390dd194f0-kube-api-access-wgvnj\") pod \"model-serving-api-fd65d7d6-l4dmh\" (UID: \"8ed463a4-8399-44b5-b2e1-8d390dd194f0\") " pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:23.285751 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.285647 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed463a4-8399-44b5-b2e1-8d390dd194f0-tls-certs\") pod \"model-serving-api-fd65d7d6-l4dmh\" (UID: \"8ed463a4-8399-44b5-b2e1-8d390dd194f0\") " pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:23.386810 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.386784 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed463a4-8399-44b5-b2e1-8d390dd194f0-tls-certs\") pod \"model-serving-api-fd65d7d6-l4dmh\" (UID: \"8ed463a4-8399-44b5-b2e1-8d390dd194f0\") " pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:23.387125 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.386963 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvnj\" (UniqueName: \"kubernetes.io/projected/8ed463a4-8399-44b5-b2e1-8d390dd194f0-kube-api-access-wgvnj\") pod \"model-serving-api-fd65d7d6-l4dmh\" (UID: \"8ed463a4-8399-44b5-b2e1-8d390dd194f0\") " pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:23.387125 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:37:23.387032 2552 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 02 18:37:23.387125 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.387060 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d95w\" (UniqueName: \"kubernetes.io/projected/b2246ded-1c56-4a1f-8348-86ff9c3941d2-kube-api-access-8d95w\") pod \"odh-model-controller-84b54b7f97-nvr8r\" (UID: \"b2246ded-1c56-4a1f-8348-86ff9c3941d2\") " pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:23.387125 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:37:23.387109 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed463a4-8399-44b5-b2e1-8d390dd194f0-tls-certs podName:8ed463a4-8399-44b5-b2e1-8d390dd194f0 nodeName:}" failed. No retries permitted until 2026-07-02 18:37:23.887087774 +0000 UTC m=+401.173437598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8ed463a4-8399-44b5-b2e1-8d390dd194f0-tls-certs") pod "model-serving-api-fd65d7d6-l4dmh" (UID: "8ed463a4-8399-44b5-b2e1-8d390dd194f0") : secret "model-serving-api-tls" not found Jul 02 18:37:23.387381 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.387299 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2246ded-1c56-4a1f-8348-86ff9c3941d2-cert\") pod \"odh-model-controller-84b54b7f97-nvr8r\" (UID: \"b2246ded-1c56-4a1f-8348-86ff9c3941d2\") " pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:23.394725 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.394697 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvnj\" (UniqueName: \"kubernetes.io/projected/8ed463a4-8399-44b5-b2e1-8d390dd194f0-kube-api-access-wgvnj\") pod \"model-serving-api-fd65d7d6-l4dmh\" (UID: \"8ed463a4-8399-44b5-b2e1-8d390dd194f0\") " pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:23.487877 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.487850 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d95w\" (UniqueName: \"kubernetes.io/projected/b2246ded-1c56-4a1f-8348-86ff9c3941d2-kube-api-access-8d95w\") pod \"odh-model-controller-84b54b7f97-nvr8r\" (UID: \"b2246ded-1c56-4a1f-8348-86ff9c3941d2\") " pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:23.487993 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.487897 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2246ded-1c56-4a1f-8348-86ff9c3941d2-cert\") pod \"odh-model-controller-84b54b7f97-nvr8r\" (UID: \"b2246ded-1c56-4a1f-8348-86ff9c3941d2\") " pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:23.490108 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.490089 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2246ded-1c56-4a1f-8348-86ff9c3941d2-cert\") pod \"odh-model-controller-84b54b7f97-nvr8r\" (UID: \"b2246ded-1c56-4a1f-8348-86ff9c3941d2\") " pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:23.495195 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.495168 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d95w\" (UniqueName: \"kubernetes.io/projected/b2246ded-1c56-4a1f-8348-86ff9c3941d2-kube-api-access-8d95w\") pod \"odh-model-controller-84b54b7f97-nvr8r\" (UID: \"b2246ded-1c56-4a1f-8348-86ff9c3941d2\") " pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:23.505111 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.505093 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:23.648498 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.648470 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-nvr8r"] Jul 02 18:37:23.651901 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:37:23.651875 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2246ded_1c56_4a1f_8348_86ff9c3941d2.slice/crio-b4206fd4446e78b731c525843454f2a8e124c65c6a5ec49a0fe09dffb6943239 WatchSource:0}: Error finding container b4206fd4446e78b731c525843454f2a8e124c65c6a5ec49a0fe09dffb6943239: Status 404 returned error can't find the container with id b4206fd4446e78b731c525843454f2a8e124c65c6a5ec49a0fe09dffb6943239 Jul 02 18:37:23.900796 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.900717 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed463a4-8399-44b5-b2e1-8d390dd194f0-tls-certs\") pod \"model-serving-api-fd65d7d6-l4dmh\" (UID: \"8ed463a4-8399-44b5-b2e1-8d390dd194f0\") " pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:23.903048 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:23.903022 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed463a4-8399-44b5-b2e1-8d390dd194f0-tls-certs\") pod \"model-serving-api-fd65d7d6-l4dmh\" (UID: \"8ed463a4-8399-44b5-b2e1-8d390dd194f0\") " pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:24.086583 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:24.086550 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:24.163155 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:24.163081 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-nvr8r" event={"ID":"b2246ded-1c56-4a1f-8348-86ff9c3941d2","Type":"ContainerStarted","Data":"b4206fd4446e78b731c525843454f2a8e124c65c6a5ec49a0fe09dffb6943239"} Jul 02 18:37:24.167931 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:24.167905 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b858d7ddb-qmprg" Jul 02 18:37:24.210173 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:24.210145 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57dd756d5f-2tjpw"] Jul 02 18:37:24.216323 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:24.216298 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-l4dmh"] Jul 02 18:37:24.218795 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:37:24.218772 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed463a4_8399_44b5_b2e1_8d390dd194f0.slice/crio-eea347dd52065f53c39feba5c0b6b4b764bf5491b093b54c315e91b25ae49afe WatchSource:0}: Error finding container eea347dd52065f53c39feba5c0b6b4b764bf5491b093b54c315e91b25ae49afe: Status 404 returned error can't find the container with id eea347dd52065f53c39feba5c0b6b4b764bf5491b093b54c315e91b25ae49afe Jul 02 18:37:25.171313 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:25.171260 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-l4dmh" event={"ID":"8ed463a4-8399-44b5-b2e1-8d390dd194f0","Type":"ContainerStarted","Data":"eea347dd52065f53c39feba5c0b6b4b764bf5491b093b54c315e91b25ae49afe"} Jul 02 18:37:27.184105 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:27.184066 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-l4dmh" event={"ID":"8ed463a4-8399-44b5-b2e1-8d390dd194f0","Type":"ContainerStarted","Data":"8838d60a0e30f09540bf4f36103f328113018f314daef47ffe2b45390395c13e"} Jul 02 18:37:27.184548 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:27.184134 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:27.185684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:27.185661 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-nvr8r" event={"ID":"b2246ded-1c56-4a1f-8348-86ff9c3941d2","Type":"ContainerStarted","Data":"bedf0782a8f7c782b0222fe34040400485572c0df5d51fef066448a6cbcdf969"} Jul 02 18:37:27.185806 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:27.185734 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:27.199876 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:27.199823 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-l4dmh" podStartSLOduration=1.4066334010000001 podStartE2EDuration="4.199805955s" totalImagesPullingTime="2.793172554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:37:27.198194893 +0000 UTC m=+404.484544736" watchObservedRunningTime="2026-07-02 18:37:27.199805955 +0000 UTC m=+404.486155799" Jul 02 18:37:27.212321 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:27.212268 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-nvr8r" podStartSLOduration=0.849735546 podStartE2EDuration="4.212250046s" totalImagesPullingTime="3.3625145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:37:27.211718045 +0000 UTC m=+404.498067889" watchObservedRunningTime="2026-07-02 18:37:27.212250046 +0000 UTC m=+404.498599890" Jul 02 18:37:38.193093 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:38.193065 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-nvr8r" Jul 02 18:37:38.195507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:38.195482 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-l4dmh" Jul 02 18:37:49.235752 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.235700 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-57dd756d5f-2tjpw" podUID="e8c5a11f-e665-4692-9262-a1fcb71f9104" containerName="console" containerID="cri-o://9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565" gracePeriod=15 Jul 02 18:37:49.475840 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.475815 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:37:49.618609 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.618525 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-trusted-ca-bundle\") pod \"e8c5a11f-e665-4692-9262-a1fcb71f9104\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " Jul 02 18:37:49.618609 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.618558 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-oauth-serving-cert\") pod \"e8c5a11f-e665-4692-9262-a1fcb71f9104\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " Jul 02 18:37:49.618830 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.618615 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-service-ca\") pod \"e8c5a11f-e665-4692-9262-a1fcb71f9104\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " Jul 02 18:37:49.618830 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.618674 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkrf7\" (UniqueName: \"kubernetes.io/projected/e8c5a11f-e665-4692-9262-a1fcb71f9104-kube-api-access-fkrf7\") pod \"e8c5a11f-e665-4692-9262-a1fcb71f9104\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " Jul 02 18:37:49.618830 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.618697 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-config\") pod \"e8c5a11f-e665-4692-9262-a1fcb71f9104\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " Jul 02 18:37:49.618830 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.618713 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-oauth-config\") pod \"e8c5a11f-e665-4692-9262-a1fcb71f9104\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " Jul 02 18:37:49.618830 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.618753 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-serving-cert\") pod \"e8c5a11f-e665-4692-9262-a1fcb71f9104\" (UID: \"e8c5a11f-e665-4692-9262-a1fcb71f9104\") " Jul 02 18:37:49.619121 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.619082 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e8c5a11f-e665-4692-9262-a1fcb71f9104" (UID: "e8c5a11f-e665-4692-9262-a1fcb71f9104"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:37:49.619121 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.619103 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-service-ca" (OuterVolumeSpecName: "service-ca") pod "e8c5a11f-e665-4692-9262-a1fcb71f9104" (UID: "e8c5a11f-e665-4692-9262-a1fcb71f9104"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:37:49.619315 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.619128 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-config" (OuterVolumeSpecName: "console-config") pod "e8c5a11f-e665-4692-9262-a1fcb71f9104" (UID: "e8c5a11f-e665-4692-9262-a1fcb71f9104"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:37:49.619315 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.619234 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e8c5a11f-e665-4692-9262-a1fcb71f9104" (UID: "e8c5a11f-e665-4692-9262-a1fcb71f9104"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:37:49.620935 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.620912 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e8c5a11f-e665-4692-9262-a1fcb71f9104" (UID: "e8c5a11f-e665-4692-9262-a1fcb71f9104"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:37:49.621022 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.620967 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e8c5a11f-e665-4692-9262-a1fcb71f9104" (UID: "e8c5a11f-e665-4692-9262-a1fcb71f9104"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:37:49.621022 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.621001 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c5a11f-e665-4692-9262-a1fcb71f9104-kube-api-access-fkrf7" (OuterVolumeSpecName: "kube-api-access-fkrf7") pod "e8c5a11f-e665-4692-9262-a1fcb71f9104" (UID: "e8c5a11f-e665-4692-9262-a1fcb71f9104"). InnerVolumeSpecName "kube-api-access-fkrf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:37:49.719807 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.719776 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:37:49.719807 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.719800 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-trusted-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:37:49.719807 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.719809 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-oauth-serving-cert\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:37:49.720025 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.719818 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-service-ca\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:37:49.720025 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.719827 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkrf7\" (UniqueName: \"kubernetes.io/projected/e8c5a11f-e665-4692-9262-a1fcb71f9104-kube-api-access-fkrf7\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:37:49.720025 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.719837 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:37:49.720025 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:49.719845 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8c5a11f-e665-4692-9262-a1fcb71f9104-console-oauth-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:37:50.287767 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:50.287734 2552 generic.go:356] "Generic (PLEG): container finished" podID="e8c5a11f-e665-4692-9262-a1fcb71f9104" containerID="9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565" exitCode=0 Jul 02 18:37:50.288192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:50.287779 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57dd756d5f-2tjpw" Jul 02 18:37:50.288192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:50.287810 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57dd756d5f-2tjpw" event={"ID":"e8c5a11f-e665-4692-9262-a1fcb71f9104","Type":"ContainerDied","Data":"9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565"} Jul 02 18:37:50.288192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:50.287849 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57dd756d5f-2tjpw" event={"ID":"e8c5a11f-e665-4692-9262-a1fcb71f9104","Type":"ContainerDied","Data":"1f5cb0ac05d2cd3a60ab4481f8e4d1c484f4944299a9eeb238f1d114d1dbbddb"} Jul 02 18:37:50.288192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:50.287869 2552 scope.go:117] "RemoveContainer" containerID="9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565" Jul 02 18:37:50.297371 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:50.297354 2552 scope.go:117] "RemoveContainer" containerID="9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565" Jul 02 18:37:50.297628 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:37:50.297611 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565\": container with ID starting with 9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565 not found: ID does not exist" containerID="9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565" Jul 02 18:37:50.297702 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:50.297683 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565"} err="failed to get container status \"9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565\": rpc error: code = NotFound desc = could not find container \"9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565\": container with ID starting with 9c1a2dfccb01be4f0ee9e2e263600dfebda6f7f921512c773c00034610c9e565 not found: ID does not exist" Jul 02 18:37:50.309163 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:50.309140 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57dd756d5f-2tjpw"] Jul 02 18:37:50.310619 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:50.310602 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57dd756d5f-2tjpw"] Jul 02 18:37:51.237075 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:37:51.237040 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c5a11f-e665-4692-9262-a1fcb71f9104" path="/var/lib/kubelet/pods/e8c5a11f-e665-4692-9262-a1fcb71f9104/volumes" Jul 02 18:39:11.510504 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.510473 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf"] Jul 02 18:39:11.510999 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.510944 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c5a11f-e665-4692-9262-a1fcb71f9104" containerName="console" Jul 02 18:39:11.510999 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.510963 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c5a11f-e665-4692-9262-a1fcb71f9104" containerName="console" Jul 02 18:39:11.511110 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.511048 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c5a11f-e665-4692-9262-a1fcb71f9104" containerName="console" Jul 02 18:39:11.513276 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.513258 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.515070 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.515049 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 02 18:39:11.515195 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.515131 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bb242-predictor-serving-cert\"" Jul 02 18:39:11.515258 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.515201 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 02 18:39:11.515258 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.515212 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bb242-kube-rbac-proxy-sar-config\"" Jul 02 18:39:11.515344 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.515201 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-t4t2c\"" Jul 02 18:39:11.522244 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.522216 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf"] Jul 02 18:39:11.705747 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.705708 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bgg\" (UniqueName: \"kubernetes.io/projected/46ca39a8-4059-437d-ac65-46df16fd1679-kube-api-access-46bgg\") pod \"success-200-isvc-bb242-predictor-86b56f46d9-98smf\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.705910 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.705778 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ca39a8-4059-437d-ac65-46df16fd1679-proxy-tls\") pod \"success-200-isvc-bb242-predictor-86b56f46d9-98smf\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.705910 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.705815 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ca39a8-4059-437d-ac65-46df16fd1679-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb242-predictor-86b56f46d9-98smf\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.807242 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.807152 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ca39a8-4059-437d-ac65-46df16fd1679-proxy-tls\") pod \"success-200-isvc-bb242-predictor-86b56f46d9-98smf\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.807242 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.807220 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ca39a8-4059-437d-ac65-46df16fd1679-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb242-predictor-86b56f46d9-98smf\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.807444 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.807272 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bgg\" (UniqueName: \"kubernetes.io/projected/46ca39a8-4059-437d-ac65-46df16fd1679-kube-api-access-46bgg\") pod \"success-200-isvc-bb242-predictor-86b56f46d9-98smf\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.807884 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.807859 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ca39a8-4059-437d-ac65-46df16fd1679-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb242-predictor-86b56f46d9-98smf\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.809600 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.809579 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ca39a8-4059-437d-ac65-46df16fd1679-proxy-tls\") pod \"success-200-isvc-bb242-predictor-86b56f46d9-98smf\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.814980 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.814960 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bgg\" (UniqueName: \"kubernetes.io/projected/46ca39a8-4059-437d-ac65-46df16fd1679-kube-api-access-46bgg\") pod \"success-200-isvc-bb242-predictor-86b56f46d9-98smf\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.823982 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.823959 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:11.961885 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:11.961713 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf"] Jul 02 18:39:11.965302 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:39:11.965274 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ca39a8_4059_437d_ac65_46df16fd1679.slice/crio-17564b25b5e72928e65a0a20b6b25db4c569dc0f50e02c8be59d4f8ffb33f6dc WatchSource:0}: Error finding container 17564b25b5e72928e65a0a20b6b25db4c569dc0f50e02c8be59d4f8ffb33f6dc: Status 404 returned error can't find the container with id 17564b25b5e72928e65a0a20b6b25db4c569dc0f50e02c8be59d4f8ffb33f6dc Jul 02 18:39:12.665394 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:12.665352 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" event={"ID":"46ca39a8-4059-437d-ac65-46df16fd1679","Type":"ContainerStarted","Data":"17564b25b5e72928e65a0a20b6b25db4c569dc0f50e02c8be59d4f8ffb33f6dc"} Jul 02 18:39:26.755298 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:26.755259 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" event={"ID":"46ca39a8-4059-437d-ac65-46df16fd1679","Type":"ContainerStarted","Data":"70427a95de5e21bd62cf2798b3e3cf667442df6784769b9d1176edfe63945277"} Jul 02 18:39:28.767462 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:28.767414 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" event={"ID":"46ca39a8-4059-437d-ac65-46df16fd1679","Type":"ContainerStarted","Data":"4053d2610cf406d8aaab5c6ae7f1353357f39621791f4480e823dc31cce6e853"} Jul 02 18:39:28.767939 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:28.767534 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:28.767939 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:28.767693 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:28.769082 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:28.769053 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 02 18:39:28.784425 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:28.784323 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podStartSLOduration=1.465261028 podStartE2EDuration="17.784301051s" totalImagesPullingTime="16.319040023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:39:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:39:28.783945646 +0000 UTC m=+526.070295500" watchObservedRunningTime="2026-07-02 18:39:28.784301051 +0000 UTC m=+526.070650898" Jul 02 18:39:29.772830 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:29.772786 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 02 18:39:34.777216 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:34.777136 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:39:34.777751 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:34.777725 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 02 18:39:44.777897 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:44.777848 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 02 18:39:54.778474 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:39:54.778437 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 02 18:40:04.778282 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:04.778232 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 02 18:40:14.778459 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:14.778432 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:40:41.591374 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.591340 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf"] Jul 02 18:40:41.593945 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.593924 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:41.595925 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.595903 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bb242-kube-rbac-proxy-sar-config\"" Jul 02 18:40:41.596021 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.595920 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bb242-serving-cert\"" Jul 02 18:40:41.604326 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.604300 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf"] Jul 02 18:40:41.701282 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.701249 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-openshift-service-ca-bundle\") pod \"switch-graph-bb242-85cccb76bd-59zpf\" (UID: \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\") " pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:41.701448 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.701299 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-proxy-tls\") pod \"switch-graph-bb242-85cccb76bd-59zpf\" (UID: \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\") " pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:41.801797 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.801759 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-openshift-service-ca-bundle\") pod \"switch-graph-bb242-85cccb76bd-59zpf\" (UID: \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\") " pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:41.801958 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.801812 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-proxy-tls\") pod \"switch-graph-bb242-85cccb76bd-59zpf\" (UID: \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\") " pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:41.802462 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.802437 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-openshift-service-ca-bundle\") pod \"switch-graph-bb242-85cccb76bd-59zpf\" (UID: \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\") " pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:41.804245 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.804218 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-proxy-tls\") pod \"switch-graph-bb242-85cccb76bd-59zpf\" (UID: \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\") " pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:41.905293 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:41.905211 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:42.033792 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:42.033768 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf"] Jul 02 18:40:42.036167 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:40:42.036129 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e2945e_e7d0_4c90_8f2c_0d0e46537827.slice/crio-5e0246032186a176651dad6d80adf61ce42c80be8af1d365785b913839b29d2f WatchSource:0}: Error finding container 5e0246032186a176651dad6d80adf61ce42c80be8af1d365785b913839b29d2f: Status 404 returned error can't find the container with id 5e0246032186a176651dad6d80adf61ce42c80be8af1d365785b913839b29d2f Jul 02 18:40:42.103989 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:42.103957 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" event={"ID":"f5e2945e-e7d0-4c90-8f2c-0d0e46537827","Type":"ContainerStarted","Data":"5e0246032186a176651dad6d80adf61ce42c80be8af1d365785b913839b29d2f"} Jul 02 18:40:45.120397 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:45.120360 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" event={"ID":"f5e2945e-e7d0-4c90-8f2c-0d0e46537827","Type":"ContainerStarted","Data":"11a76194d0fb981a14de97aff279ca378eee37ca25642563cfd32bf8e1144e6b"} Jul 02 18:40:45.120844 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:45.120444 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:45.135780 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:45.135732 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" podStartSLOduration=1.633363456 podStartE2EDuration="4.135717392s" totalImagesPullingTime="2.502353936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:40:45.13428735 +0000 UTC m=+602.420637190" watchObservedRunningTime="2026-07-02 18:40:45.135717392 +0000 UTC m=+602.422067234" Jul 02 18:40:51.130926 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.130893 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:40:51.732789 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.732756 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf"] Jul 02 18:40:51.732969 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.732931 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" containerID="cri-o://11a76194d0fb981a14de97aff279ca378eee37ca25642563cfd32bf8e1144e6b" gracePeriod=30 Jul 02 18:40:51.916585 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.916520 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6"] Jul 02 18:40:51.919952 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.919932 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:51.921832 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.921810 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a5373-predictor-serving-cert\"" Jul 02 18:40:51.921832 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.921822 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a5373-kube-rbac-proxy-sar-config\"" Jul 02 18:40:51.927445 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.927413 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6"] Jul 02 18:40:51.930656 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.930610 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf"] Jul 02 18:40:51.930968 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.930935 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" containerID="cri-o://70427a95de5e21bd62cf2798b3e3cf667442df6784769b9d1176edfe63945277" gracePeriod=30 Jul 02 18:40:51.931090 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.930991 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kube-rbac-proxy" containerID="cri-o://4053d2610cf406d8aaab5c6ae7f1353357f39621791f4480e823dc31cce6e853" gracePeriod=30 Jul 02 18:40:51.986491 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.986417 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3569485e-42cc-47cd-9239-125860130e89-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:51.986491 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.986468 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3569485e-42cc-47cd-9239-125860130e89-proxy-tls\") pod \"success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:51.986684 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:51.986576 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxczp\" (UniqueName: \"kubernetes.io/projected/3569485e-42cc-47cd-9239-125860130e89-kube-api-access-hxczp\") pod \"success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:52.087446 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.087412 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxczp\" (UniqueName: \"kubernetes.io/projected/3569485e-42cc-47cd-9239-125860130e89-kube-api-access-hxczp\") pod \"success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:52.087603 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.087449 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3569485e-42cc-47cd-9239-125860130e89-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:52.087603 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.087536 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3569485e-42cc-47cd-9239-125860130e89-proxy-tls\") pod \"success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:52.088173 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.088147 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3569485e-42cc-47cd-9239-125860130e89-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:52.089976 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.089955 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3569485e-42cc-47cd-9239-125860130e89-proxy-tls\") pod \"success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:52.096575 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.096532 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxczp\" (UniqueName: \"kubernetes.io/projected/3569485e-42cc-47cd-9239-125860130e89-kube-api-access-hxczp\") pod \"success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:52.159727 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.159693 2552 generic.go:356] "Generic (PLEG): container finished" podID="46ca39a8-4059-437d-ac65-46df16fd1679" containerID="4053d2610cf406d8aaab5c6ae7f1353357f39621791f4480e823dc31cce6e853" exitCode=2 Jul 02 18:40:52.160084 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.159749 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" event={"ID":"46ca39a8-4059-437d-ac65-46df16fd1679","Type":"ContainerDied","Data":"4053d2610cf406d8aaab5c6ae7f1353357f39621791f4480e823dc31cce6e853"} Jul 02 18:40:52.235365 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.235285 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:52.363446 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:52.363309 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6"] Jul 02 18:40:52.366135 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:40:52.366108 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3569485e_42cc_47cd_9239_125860130e89.slice/crio-1d8550989a7c29f6a4b0db20d37502ba5e04bf0da70e1f9729d430f024f9076e WatchSource:0}: Error finding container 1d8550989a7c29f6a4b0db20d37502ba5e04bf0da70e1f9729d430f024f9076e: Status 404 returned error can't find the container with id 1d8550989a7c29f6a4b0db20d37502ba5e04bf0da70e1f9729d430f024f9076e Jul 02 18:40:53.173772 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:53.173729 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" event={"ID":"3569485e-42cc-47cd-9239-125860130e89","Type":"ContainerStarted","Data":"cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f"} Jul 02 18:40:53.174216 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:53.173814 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" event={"ID":"3569485e-42cc-47cd-9239-125860130e89","Type":"ContainerStarted","Data":"677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8"} Jul 02 18:40:53.174216 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:53.173841 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:53.174216 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:53.173860 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" event={"ID":"3569485e-42cc-47cd-9239-125860130e89","Type":"ContainerStarted","Data":"1d8550989a7c29f6a4b0db20d37502ba5e04bf0da70e1f9729d430f024f9076e"} Jul 02 18:40:53.174216 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:53.173879 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:53.175344 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:53.175318 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 02 18:40:53.191409 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:53.191359 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podStartSLOduration=2.191346055 podStartE2EDuration="2.191346055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:40:53.190172917 +0000 UTC m=+610.476522771" watchObservedRunningTime="2026-07-02 18:40:53.191346055 +0000 UTC m=+610.477695897" Jul 02 18:40:54.178147 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:54.178107 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 02 18:40:54.774059 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:54.774008 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused" Jul 02 18:40:54.778201 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:54.778176 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 02 18:40:55.184682 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.184651 2552 generic.go:356] "Generic (PLEG): container finished" podID="46ca39a8-4059-437d-ac65-46df16fd1679" containerID="70427a95de5e21bd62cf2798b3e3cf667442df6784769b9d1176edfe63945277" exitCode=0 Jul 02 18:40:55.185077 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.184706 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" event={"ID":"46ca39a8-4059-437d-ac65-46df16fd1679","Type":"ContainerDied","Data":"70427a95de5e21bd62cf2798b3e3cf667442df6784769b9d1176edfe63945277"} Jul 02 18:40:55.288722 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.288696 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:40:55.410339 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.410239 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ca39a8-4059-437d-ac65-46df16fd1679-proxy-tls\") pod \"46ca39a8-4059-437d-ac65-46df16fd1679\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " Jul 02 18:40:55.410339 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.410339 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46bgg\" (UniqueName: \"kubernetes.io/projected/46ca39a8-4059-437d-ac65-46df16fd1679-kube-api-access-46bgg\") pod \"46ca39a8-4059-437d-ac65-46df16fd1679\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " Jul 02 18:40:55.410567 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.410358 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ca39a8-4059-437d-ac65-46df16fd1679-kube-rbac-proxy-sar-config\") pod \"46ca39a8-4059-437d-ac65-46df16fd1679\" (UID: \"46ca39a8-4059-437d-ac65-46df16fd1679\") " Jul 02 18:40:55.410819 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.410785 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ca39a8-4059-437d-ac65-46df16fd1679-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "46ca39a8-4059-437d-ac65-46df16fd1679" (UID: "46ca39a8-4059-437d-ac65-46df16fd1679"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:40:55.412429 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.412409 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ca39a8-4059-437d-ac65-46df16fd1679-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46ca39a8-4059-437d-ac65-46df16fd1679" (UID: "46ca39a8-4059-437d-ac65-46df16fd1679"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:40:55.412528 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.412475 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ca39a8-4059-437d-ac65-46df16fd1679-kube-api-access-46bgg" (OuterVolumeSpecName: "kube-api-access-46bgg") pod "46ca39a8-4059-437d-ac65-46df16fd1679" (UID: "46ca39a8-4059-437d-ac65-46df16fd1679"). InnerVolumeSpecName "kube-api-access-46bgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:40:55.510905 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.510868 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ca39a8-4059-437d-ac65-46df16fd1679-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:40:55.510905 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.510898 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bgg\" (UniqueName: \"kubernetes.io/projected/46ca39a8-4059-437d-ac65-46df16fd1679-kube-api-access-46bgg\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:40:55.510905 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:55.510910 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ca39a8-4059-437d-ac65-46df16fd1679-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:40:56.129598 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:56.129556 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:40:56.191277 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:56.191245 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" event={"ID":"46ca39a8-4059-437d-ac65-46df16fd1679","Type":"ContainerDied","Data":"17564b25b5e72928e65a0a20b6b25db4c569dc0f50e02c8be59d4f8ffb33f6dc"} Jul 02 18:40:56.191700 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:56.191282 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf" Jul 02 18:40:56.191700 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:56.191314 2552 scope.go:117] "RemoveContainer" containerID="4053d2610cf406d8aaab5c6ae7f1353357f39621791f4480e823dc31cce6e853" Jul 02 18:40:56.200533 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:56.200513 2552 scope.go:117] "RemoveContainer" containerID="70427a95de5e21bd62cf2798b3e3cf667442df6784769b9d1176edfe63945277" Jul 02 18:40:56.211420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:56.211399 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf"] Jul 02 18:40:56.215220 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:56.215189 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb242-predictor-86b56f46d9-98smf"] Jul 02 18:40:57.236170 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:57.236135 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" path="/var/lib/kubelet/pods/46ca39a8-4059-437d-ac65-46df16fd1679/volumes" Jul 02 18:40:59.183362 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:59.183334 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:40:59.183824 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:40:59.183798 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 02 18:41:01.129347 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:01.129309 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:41:06.129122 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:06.129036 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:41:06.129588 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:06.129163 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:41:09.184460 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:09.184421 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 02 18:41:11.129905 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:11.129866 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:41:16.129858 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:16.129820 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:41:19.184803 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:19.184756 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 02 18:41:21.129652 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:21.129590 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:41:22.313784 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:22.313740 2552 generic.go:356] "Generic (PLEG): container finished" podID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerID="11a76194d0fb981a14de97aff279ca378eee37ca25642563cfd32bf8e1144e6b" exitCode=0 Jul 02 18:41:22.314227 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:22.313977 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" event={"ID":"f5e2945e-e7d0-4c90-8f2c-0d0e46537827","Type":"ContainerDied","Data":"11a76194d0fb981a14de97aff279ca378eee37ca25642563cfd32bf8e1144e6b"} Jul 02 18:41:22.387990 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:22.387963 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:41:22.452036 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:22.451997 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-proxy-tls\") pod \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\" (UID: \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\") " Jul 02 18:41:22.452201 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:22.452069 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-openshift-service-ca-bundle\") pod \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\" (UID: \"f5e2945e-e7d0-4c90-8f2c-0d0e46537827\") " Jul 02 18:41:22.452403 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:22.452382 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f5e2945e-e7d0-4c90-8f2c-0d0e46537827" (UID: "f5e2945e-e7d0-4c90-8f2c-0d0e46537827"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:41:22.454071 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:22.454040 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5e2945e-e7d0-4c90-8f2c-0d0e46537827" (UID: "f5e2945e-e7d0-4c90-8f2c-0d0e46537827"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:41:22.552890 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:22.552799 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:41:22.552890 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:22.552829 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5e2945e-e7d0-4c90-8f2c-0d0e46537827-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:41:23.319557 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:23.319527 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" Jul 02 18:41:23.319995 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:23.319558 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf" event={"ID":"f5e2945e-e7d0-4c90-8f2c-0d0e46537827","Type":"ContainerDied","Data":"5e0246032186a176651dad6d80adf61ce42c80be8af1d365785b913839b29d2f"} Jul 02 18:41:23.319995 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:23.319619 2552 scope.go:117] "RemoveContainer" containerID="11a76194d0fb981a14de97aff279ca378eee37ca25642563cfd32bf8e1144e6b" Jul 02 18:41:23.334434 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:23.334411 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf"] Jul 02 18:41:23.337978 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:23.337952 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb242-85cccb76bd-59zpf"] Jul 02 18:41:25.236591 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:25.236550 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" path="/var/lib/kubelet/pods/f5e2945e-e7d0-4c90-8f2c-0d0e46537827/volumes" Jul 02 18:41:29.184418 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:29.184376 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 02 18:41:31.554542 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.554508 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798"] Jul 02 18:41:31.554927 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.554894 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" Jul 02 18:41:31.554927 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.554905 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" Jul 02 18:41:31.554927 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.554926 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" Jul 02 18:41:31.555046 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.554933 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" Jul 02 18:41:31.555046 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.554940 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kube-rbac-proxy" Jul 02 18:41:31.555046 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.554946 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kube-rbac-proxy" Jul 02 18:41:31.555046 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.555001 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kserve-container" Jul 02 18:41:31.555046 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.555013 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e2945e-e7d0-4c90-8f2c-0d0e46537827" containerName="switch-graph-bb242" Jul 02 18:41:31.555046 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.555024 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ca39a8-4059-437d-ac65-46df16fd1679" containerName="kube-rbac-proxy" Jul 02 18:41:31.559552 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.559534 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:31.561421 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.561403 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 02 18:41:31.561521 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.561453 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 02 18:41:31.564346 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.564318 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798"] Jul 02 18:41:31.645664 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.642680 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73797b34-dc9c-4c45-9ecc-959e27908789-openshift-service-ca-bundle\") pod \"model-chainer-5bd77c7cd9-6j798\" (UID: \"73797b34-dc9c-4c45-9ecc-959e27908789\") " pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:31.645664 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.642775 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73797b34-dc9c-4c45-9ecc-959e27908789-proxy-tls\") pod \"model-chainer-5bd77c7cd9-6j798\" (UID: \"73797b34-dc9c-4c45-9ecc-959e27908789\") " pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:31.743461 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.743422 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73797b34-dc9c-4c45-9ecc-959e27908789-openshift-service-ca-bundle\") pod \"model-chainer-5bd77c7cd9-6j798\" (UID: \"73797b34-dc9c-4c45-9ecc-959e27908789\") " pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:31.743664 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.743474 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73797b34-dc9c-4c45-9ecc-959e27908789-proxy-tls\") pod \"model-chainer-5bd77c7cd9-6j798\" (UID: \"73797b34-dc9c-4c45-9ecc-959e27908789\") " pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:31.743664 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:41:31.743605 2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Jul 02 18:41:31.743742 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:41:31.743680 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73797b34-dc9c-4c45-9ecc-959e27908789-proxy-tls podName:73797b34-dc9c-4c45-9ecc-959e27908789 nodeName:}" failed. No retries permitted until 2026-07-02 18:41:32.243656635 +0000 UTC m=+649.530006455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/73797b34-dc9c-4c45-9ecc-959e27908789-proxy-tls") pod "model-chainer-5bd77c7cd9-6j798" (UID: "73797b34-dc9c-4c45-9ecc-959e27908789") : secret "model-chainer-serving-cert" not found Jul 02 18:41:31.744098 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:31.744075 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73797b34-dc9c-4c45-9ecc-959e27908789-openshift-service-ca-bundle\") pod \"model-chainer-5bd77c7cd9-6j798\" (UID: \"73797b34-dc9c-4c45-9ecc-959e27908789\") " pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:32.247699 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:32.247654 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73797b34-dc9c-4c45-9ecc-959e27908789-proxy-tls\") pod \"model-chainer-5bd77c7cd9-6j798\" (UID: \"73797b34-dc9c-4c45-9ecc-959e27908789\") " pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:32.250025 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:32.250006 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73797b34-dc9c-4c45-9ecc-959e27908789-proxy-tls\") pod \"model-chainer-5bd77c7cd9-6j798\" (UID: \"73797b34-dc9c-4c45-9ecc-959e27908789\") " pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:32.471875 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:32.471835 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:32.602330 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:32.602288 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798"] Jul 02 18:41:32.604692 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:41:32.604664 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73797b34_dc9c_4c45_9ecc_959e27908789.slice/crio-3e8850fea4047f349f2c4a1a1e4dadc1e650be7765933e89b42d3df416e67647 WatchSource:0}: Error finding container 3e8850fea4047f349f2c4a1a1e4dadc1e650be7765933e89b42d3df416e67647: Status 404 returned error can't find the container with id 3e8850fea4047f349f2c4a1a1e4dadc1e650be7765933e89b42d3df416e67647 Jul 02 18:41:33.364557 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:33.364508 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" event={"ID":"73797b34-dc9c-4c45-9ecc-959e27908789","Type":"ContainerStarted","Data":"83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89"} Jul 02 18:41:33.364743 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:33.364573 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" event={"ID":"73797b34-dc9c-4c45-9ecc-959e27908789","Type":"ContainerStarted","Data":"3e8850fea4047f349f2c4a1a1e4dadc1e650be7765933e89b42d3df416e67647"} Jul 02 18:41:33.364743 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:33.364600 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:33.379969 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:33.379917 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" podStartSLOduration=2.37990094 podStartE2EDuration="2.37990094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:41:33.378929034 +0000 UTC m=+650.665278876" watchObservedRunningTime="2026-07-02 18:41:33.37990094 +0000 UTC m=+650.666250782" Jul 02 18:41:39.184859 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:39.184827 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:41:39.375901 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:39.375873 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:41.657887 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.657851 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798"] Jul 02 18:41:41.658368 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.658064 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" containerID="cri-o://83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89" gracePeriod=30 Jul 02 18:41:41.820437 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.820404 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d"] Jul 02 18:41:41.824889 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.824861 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:41.826934 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.826906 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e8b76-predictor-serving-cert\"" Jul 02 18:41:41.827081 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.826949 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e8b76-kube-rbac-proxy-sar-config\"" Jul 02 18:41:41.835313 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.835220 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d"] Jul 02 18:41:41.932100 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.932070 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23f82c5-0e33-4821-a596-783a6938c9db-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:41.932273 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.932123 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkgqc\" (UniqueName: \"kubernetes.io/projected/d23f82c5-0e33-4821-a596-783a6938c9db-kube-api-access-bkgqc\") pod \"success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:41.932273 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:41.932149 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23f82c5-0e33-4821-a596-783a6938c9db-proxy-tls\") pod \"success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:42.032706 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.032668 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkgqc\" (UniqueName: \"kubernetes.io/projected/d23f82c5-0e33-4821-a596-783a6938c9db-kube-api-access-bkgqc\") pod \"success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:42.032706 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.032715 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23f82c5-0e33-4821-a596-783a6938c9db-proxy-tls\") pod \"success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:42.032958 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.032800 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23f82c5-0e33-4821-a596-783a6938c9db-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:42.033497 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.033474 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23f82c5-0e33-4821-a596-783a6938c9db-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:42.035221 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.035202 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23f82c5-0e33-4821-a596-783a6938c9db-proxy-tls\") pod \"success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:42.039519 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.039496 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkgqc\" (UniqueName: \"kubernetes.io/projected/d23f82c5-0e33-4821-a596-783a6938c9db-kube-api-access-bkgqc\") pod \"success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:42.137772 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.137732 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:42.278803 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.278780 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d"] Jul 02 18:41:42.281013 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:41:42.280989 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23f82c5_0e33_4821_a596_783a6938c9db.slice/crio-a3f545266743e5339f667d7e921bad31c241c6e741cc45ca734597e30b410bc9 WatchSource:0}: Error finding container a3f545266743e5339f667d7e921bad31c241c6e741cc45ca734597e30b410bc9: Status 404 returned error can't find the container with id a3f545266743e5339f667d7e921bad31c241c6e741cc45ca734597e30b410bc9 Jul 02 18:41:42.407592 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.407562 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" event={"ID":"d23f82c5-0e33-4821-a596-783a6938c9db","Type":"ContainerStarted","Data":"ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb"} Jul 02 18:41:42.407740 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.407620 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" event={"ID":"d23f82c5-0e33-4821-a596-783a6938c9db","Type":"ContainerStarted","Data":"0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c"} Jul 02 18:41:42.413461 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.408988 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" event={"ID":"d23f82c5-0e33-4821-a596-783a6938c9db","Type":"ContainerStarted","Data":"a3f545266743e5339f667d7e921bad31c241c6e741cc45ca734597e30b410bc9"} Jul 02 18:41:42.429417 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:42.429264 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podStartSLOduration=1.429245645 podStartE2EDuration="1.429245645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:41:42.428403891 +0000 UTC m=+659.714753724" watchObservedRunningTime="2026-07-02 18:41:42.429245645 +0000 UTC m=+659.715595492" Jul 02 18:41:43.411958 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:43.411928 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:43.412361 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:43.411986 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:43.413394 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:43.413364 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 02 18:41:44.375220 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:44.375179 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:41:44.418159 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:44.418127 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 02 18:41:49.374400 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:49.374356 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:41:49.423426 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:49.423394 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:41:49.424015 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:49.423992 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 02 18:41:51.929160 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:51.929125 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq"] Jul 02 18:41:51.941269 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:51.941237 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq"] Jul 02 18:41:51.941414 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:51.941339 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:51.943051 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:51.943027 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a5373-serving-cert\"" Jul 02 18:41:51.943191 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:51.943055 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a5373-kube-rbac-proxy-sar-config\"" Jul 02 18:41:52.013511 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:52.013464 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls\") pod \"switch-graph-a5373-79d8b76794-sl4kq\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:52.013712 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:52.013528 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fd8493-acbe-4086-b17c-f43e0381c905-openshift-service-ca-bundle\") pod \"switch-graph-a5373-79d8b76794-sl4kq\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:52.114721 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:52.114682 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fd8493-acbe-4086-b17c-f43e0381c905-openshift-service-ca-bundle\") pod \"switch-graph-a5373-79d8b76794-sl4kq\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:52.114937 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:52.114773 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls\") pod \"switch-graph-a5373-79d8b76794-sl4kq\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:52.114937 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:41:52.114889 2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-a5373-serving-cert: secret "switch-graph-a5373-serving-cert" not found Jul 02 18:41:52.115062 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:41:52.114953 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls podName:f4fd8493-acbe-4086-b17c-f43e0381c905 nodeName:}" failed. No retries permitted until 2026-07-02 18:41:52.614936955 +0000 UTC m=+669.901286776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls") pod "switch-graph-a5373-79d8b76794-sl4kq" (UID: "f4fd8493-acbe-4086-b17c-f43e0381c905") : secret "switch-graph-a5373-serving-cert" not found Jul 02 18:41:52.115434 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:52.115409 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fd8493-acbe-4086-b17c-f43e0381c905-openshift-service-ca-bundle\") pod \"switch-graph-a5373-79d8b76794-sl4kq\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:52.618969 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:52.618936 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls\") pod \"switch-graph-a5373-79d8b76794-sl4kq\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:52.619147 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:41:52.619050 2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-a5373-serving-cert: secret "switch-graph-a5373-serving-cert" not found Jul 02 18:41:52.619147 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:41:52.619103 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls podName:f4fd8493-acbe-4086-b17c-f43e0381c905 nodeName:}" failed. No retries permitted until 2026-07-02 18:41:53.619087777 +0000 UTC m=+670.905437603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls") pod "switch-graph-a5373-79d8b76794-sl4kq" (UID: "f4fd8493-acbe-4086-b17c-f43e0381c905") : secret "switch-graph-a5373-serving-cert" not found Jul 02 18:41:53.629341 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:53.629303 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls\") pod \"switch-graph-a5373-79d8b76794-sl4kq\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:53.631860 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:53.631838 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls\") pod \"switch-graph-a5373-79d8b76794-sl4kq\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:53.753078 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:53.753023 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:53.885142 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:53.885062 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq"] Jul 02 18:41:53.888628 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:41:53.888598 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fd8493_acbe_4086_b17c_f43e0381c905.slice/crio-b27241e97e2d357f44d2e7e18f2063a696464104e3fe5e6f0bdb735f81aa79f7 WatchSource:0}: Error finding container b27241e97e2d357f44d2e7e18f2063a696464104e3fe5e6f0bdb735f81aa79f7: Status 404 returned error can't find the container with id b27241e97e2d357f44d2e7e18f2063a696464104e3fe5e6f0bdb735f81aa79f7 Jul 02 18:41:54.374735 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:54.374692 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:41:54.374922 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:54.374806 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:41:54.468962 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:54.468925 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" event={"ID":"f4fd8493-acbe-4086-b17c-f43e0381c905","Type":"ContainerStarted","Data":"b7869031be38a902fe6adbfd70babc3337bba27a740e23cb7158aaeead7391a3"} Jul 02 18:41:54.469150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:54.468991 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" event={"ID":"f4fd8493-acbe-4086-b17c-f43e0381c905","Type":"ContainerStarted","Data":"b27241e97e2d357f44d2e7e18f2063a696464104e3fe5e6f0bdb735f81aa79f7"} Jul 02 18:41:54.469150 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:54.469023 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:41:54.484079 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:54.484023 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" podStartSLOduration=3.483999503 podStartE2EDuration="3.483999503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:41:54.482009314 +0000 UTC m=+671.768359160" watchObservedRunningTime="2026-07-02 18:41:54.483999503 +0000 UTC m=+671.770349345" Jul 02 18:41:59.373863 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:59.373828 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:41:59.424197 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:41:59.424159 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 02 18:42:00.481388 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:00.481361 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:42:04.374250 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:04.374207 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:42:09.374157 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:09.374111 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:42:09.424781 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:09.424739 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 02 18:42:11.807994 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:11.807965 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:42:11.887755 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:11.887719 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73797b34-dc9c-4c45-9ecc-959e27908789-proxy-tls\") pod \"73797b34-dc9c-4c45-9ecc-959e27908789\" (UID: \"73797b34-dc9c-4c45-9ecc-959e27908789\") " Jul 02 18:42:11.887910 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:11.887827 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73797b34-dc9c-4c45-9ecc-959e27908789-openshift-service-ca-bundle\") pod \"73797b34-dc9c-4c45-9ecc-959e27908789\" (UID: \"73797b34-dc9c-4c45-9ecc-959e27908789\") " Jul 02 18:42:11.888094 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:11.888067 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73797b34-dc9c-4c45-9ecc-959e27908789-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "73797b34-dc9c-4c45-9ecc-959e27908789" (UID: "73797b34-dc9c-4c45-9ecc-959e27908789"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:42:11.889788 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:11.889758 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73797b34-dc9c-4c45-9ecc-959e27908789-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73797b34-dc9c-4c45-9ecc-959e27908789" (UID: "73797b34-dc9c-4c45-9ecc-959e27908789"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:42:11.988602 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:11.988569 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73797b34-dc9c-4c45-9ecc-959e27908789-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:42:11.988602 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:11.988601 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73797b34-dc9c-4c45-9ecc-959e27908789-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:42:12.552179 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:12.552145 2552 generic.go:356] "Generic (PLEG): container finished" podID="73797b34-dc9c-4c45-9ecc-959e27908789" containerID="83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89" exitCode=0 Jul 02 18:42:12.552350 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:12.552193 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" Jul 02 18:42:12.552350 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:12.552208 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" event={"ID":"73797b34-dc9c-4c45-9ecc-959e27908789","Type":"ContainerDied","Data":"83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89"} Jul 02 18:42:12.552350 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:12.552247 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798" event={"ID":"73797b34-dc9c-4c45-9ecc-959e27908789","Type":"ContainerDied","Data":"3e8850fea4047f349f2c4a1a1e4dadc1e650be7765933e89b42d3df416e67647"} Jul 02 18:42:12.552350 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:12.552268 2552 scope.go:117] "RemoveContainer" containerID="83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89" Jul 02 18:42:12.560873 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:12.560854 2552 scope.go:117] "RemoveContainer" containerID="83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89" Jul 02 18:42:12.561149 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:42:12.561130 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89\": container with ID starting with 83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89 not found: ID does not exist" containerID="83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89" Jul 02 18:42:12.561192 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:12.561160 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89"} err="failed to get container status \"83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89\": rpc error: code = NotFound desc = could not find container \"83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89\": container with ID starting with 83dafa75798c52047cab91d68d38351a18d481f4a3bd6f9d563a37fcafd68d89 not found: ID does not exist" Jul 02 18:42:12.571922 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:12.571894 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798"] Jul 02 18:42:12.577190 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:12.577169 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5bd77c7cd9-6j798"] Jul 02 18:42:13.236726 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:13.236695 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" path="/var/lib/kubelet/pods/73797b34-dc9c-4c45-9ecc-959e27908789/volumes" Jul 02 18:42:19.424535 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:19.424497 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 02 18:42:29.424758 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:29.424728 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:42:41.845750 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.845665 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n"] Jul 02 18:42:41.846120 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.846039 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" Jul 02 18:42:41.846120 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.846050 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" Jul 02 18:42:41.846221 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.846125 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73797b34-dc9c-4c45-9ecc-959e27908789" containerName="model-chainer" Jul 02 18:42:41.849010 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.848990 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:41.850902 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.850882 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-e8b76-serving-cert\"" Jul 02 18:42:41.850986 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.850882 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-e8b76-kube-rbac-proxy-sar-config\"" Jul 02 18:42:41.856380 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.856342 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n"] Jul 02 18:42:41.960721 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.960689 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7715988c-7201-4301-9699-e351641dafbf-proxy-tls\") pod \"sequence-graph-e8b76-cc658d568-n4p7n\" (UID: \"7715988c-7201-4301-9699-e351641dafbf\") " pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:41.960898 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:41.960751 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7715988c-7201-4301-9699-e351641dafbf-openshift-service-ca-bundle\") pod \"sequence-graph-e8b76-cc658d568-n4p7n\" (UID: \"7715988c-7201-4301-9699-e351641dafbf\") " pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:42.061589 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:42.061551 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7715988c-7201-4301-9699-e351641dafbf-proxy-tls\") pod \"sequence-graph-e8b76-cc658d568-n4p7n\" (UID: \"7715988c-7201-4301-9699-e351641dafbf\") " pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:42.061764 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:42.061611 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7715988c-7201-4301-9699-e351641dafbf-openshift-service-ca-bundle\") pod \"sequence-graph-e8b76-cc658d568-n4p7n\" (UID: \"7715988c-7201-4301-9699-e351641dafbf\") " pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:42.061764 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:42:42.061723 2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-e8b76-serving-cert: secret "sequence-graph-e8b76-serving-cert" not found Jul 02 18:42:42.061857 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:42:42.061796 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7715988c-7201-4301-9699-e351641dafbf-proxy-tls podName:7715988c-7201-4301-9699-e351641dafbf nodeName:}" failed. No retries permitted until 2026-07-02 18:42:42.561777016 +0000 UTC m=+719.848126837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7715988c-7201-4301-9699-e351641dafbf-proxy-tls") pod "sequence-graph-e8b76-cc658d568-n4p7n" (UID: "7715988c-7201-4301-9699-e351641dafbf") : secret "sequence-graph-e8b76-serving-cert" not found Jul 02 18:42:42.062941 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:42.062913 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7715988c-7201-4301-9699-e351641dafbf-openshift-service-ca-bundle\") pod \"sequence-graph-e8b76-cc658d568-n4p7n\" (UID: \"7715988c-7201-4301-9699-e351641dafbf\") " pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:42.565742 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:42.565708 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7715988c-7201-4301-9699-e351641dafbf-proxy-tls\") pod \"sequence-graph-e8b76-cc658d568-n4p7n\" (UID: \"7715988c-7201-4301-9699-e351641dafbf\") " pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:42.568042 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:42.568011 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7715988c-7201-4301-9699-e351641dafbf-proxy-tls\") pod \"sequence-graph-e8b76-cc658d568-n4p7n\" (UID: \"7715988c-7201-4301-9699-e351641dafbf\") " pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:42.760315 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:42.760279 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:42.897581 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:42.897548 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n"] Jul 02 18:42:42.900687 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:42:42.900661 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7715988c_7201_4301_9699_e351641dafbf.slice/crio-0053705c55a613f9cfc9f26dee53df3a22b0b741727d0b13950021e1782a9e03 WatchSource:0}: Error finding container 0053705c55a613f9cfc9f26dee53df3a22b0b741727d0b13950021e1782a9e03: Status 404 returned error can't find the container with id 0053705c55a613f9cfc9f26dee53df3a22b0b741727d0b13950021e1782a9e03 Jul 02 18:42:43.695101 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:43.695066 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" event={"ID":"7715988c-7201-4301-9699-e351641dafbf","Type":"ContainerStarted","Data":"893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b"} Jul 02 18:42:43.695268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:43.695126 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:42:43.695268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:43.695142 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" event={"ID":"7715988c-7201-4301-9699-e351641dafbf","Type":"ContainerStarted","Data":"0053705c55a613f9cfc9f26dee53df3a22b0b741727d0b13950021e1782a9e03"} Jul 02 18:42:43.711473 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:43.711424 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" podStartSLOduration=2.71140813 podStartE2EDuration="2.71140813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:42:43.709904331 +0000 UTC m=+720.996254175" watchObservedRunningTime="2026-07-02 18:42:43.71140813 +0000 UTC m=+720.997757973" Jul 02 18:42:49.705386 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:42:49.705345 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:50:16.705848 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.705749 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq"] Jul 02 18:50:16.706346 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.705998 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" containerID="cri-o://b7869031be38a902fe6adbfd70babc3337bba27a740e23cb7158aaeead7391a3" gracePeriod=30 Jul 02 18:50:16.892204 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.892169 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv"] Jul 02 18:50:16.895911 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.895892 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:16.898892 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.898873 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e22cf-predictor-serving-cert\"" Jul 02 18:50:16.898988 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.898956 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e22cf-kube-rbac-proxy-sar-config\"" Jul 02 18:50:16.904507 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.904487 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6"] Jul 02 18:50:16.904749 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.904706 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" containerID="cri-o://677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8" gracePeriod=30 Jul 02 18:50:16.904837 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.904789 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kube-rbac-proxy" containerID="cri-o://cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f" gracePeriod=30 Jul 02 18:50:16.912697 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.912670 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv"] Jul 02 18:50:16.993432 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.993350 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcnwt\" (UniqueName: \"kubernetes.io/projected/104b64ce-542f-4113-9fea-d346c4f4991e-kube-api-access-tcnwt\") pod \"success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:16.993432 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.993427 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b64ce-542f-4113-9fea-d346c4f4991e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:16.993608 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:16.993447 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b64ce-542f-4113-9fea-d346c4f4991e-proxy-tls\") pod \"success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.094788 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.094757 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcnwt\" (UniqueName: \"kubernetes.io/projected/104b64ce-542f-4113-9fea-d346c4f4991e-kube-api-access-tcnwt\") pod \"success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.094962 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.094807 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b64ce-542f-4113-9fea-d346c4f4991e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.094962 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.094829 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b64ce-542f-4113-9fea-d346c4f4991e-proxy-tls\") pod \"success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.095415 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.095389 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b64ce-542f-4113-9fea-d346c4f4991e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.097257 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.097229 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b64ce-542f-4113-9fea-d346c4f4991e-proxy-tls\") pod \"success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.103813 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.103789 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcnwt\" (UniqueName: \"kubernetes.io/projected/104b64ce-542f-4113-9fea-d346c4f4991e-kube-api-access-tcnwt\") pod \"success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.207162 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.207127 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.338207 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.338178 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv"] Jul 02 18:50:17.340341 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:50:17.340308 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104b64ce_542f_4113_9fea_d346c4f4991e.slice/crio-98f5d2dfd285b0e6514c411adc576d0466541a292bd2747524baf46b8bc0cc83 WatchSource:0}: Error finding container 98f5d2dfd285b0e6514c411adc576d0466541a292bd2747524baf46b8bc0cc83: Status 404 returned error can't find the container with id 98f5d2dfd285b0e6514c411adc576d0466541a292bd2747524baf46b8bc0cc83 Jul 02 18:50:17.803113 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.803078 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" event={"ID":"104b64ce-542f-4113-9fea-d346c4f4991e","Type":"ContainerStarted","Data":"c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4"} Jul 02 18:50:17.803550 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.803148 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" event={"ID":"104b64ce-542f-4113-9fea-d346c4f4991e","Type":"ContainerStarted","Data":"c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76"} Jul 02 18:50:17.803550 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.803182 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" event={"ID":"104b64ce-542f-4113-9fea-d346c4f4991e","Type":"ContainerStarted","Data":"98f5d2dfd285b0e6514c411adc576d0466541a292bd2747524baf46b8bc0cc83"} Jul 02 18:50:17.803550 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.803215 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.803550 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.803248 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:17.809655 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.807013 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 02 18:50:17.810235 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.810205 2552 generic.go:356] "Generic (PLEG): container finished" podID="3569485e-42cc-47cd-9239-125860130e89" containerID="cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f" exitCode=2 Jul 02 18:50:17.810448 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.810429 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" event={"ID":"3569485e-42cc-47cd-9239-125860130e89","Type":"ContainerDied","Data":"cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f"} Jul 02 18:50:17.821053 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:17.821011 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podStartSLOduration=1.8209981370000001 podStartE2EDuration="1.820998137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:50:17.819162306 +0000 UTC m=+1175.105512150" watchObservedRunningTime="2026-07-02 18:50:17.820998137 +0000 UTC m=+1175.107347980" Jul 02 18:50:18.814854 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:18.814808 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 02 18:50:19.178629 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:19.178585 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" Jul 02 18:50:19.184567 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:19.184541 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 02 18:50:20.062903 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.062876 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:50:20.118799 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.118710 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxczp\" (UniqueName: \"kubernetes.io/projected/3569485e-42cc-47cd-9239-125860130e89-kube-api-access-hxczp\") pod \"3569485e-42cc-47cd-9239-125860130e89\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " Jul 02 18:50:20.118799 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.118786 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3569485e-42cc-47cd-9239-125860130e89-kube-rbac-proxy-sar-config\") pod \"3569485e-42cc-47cd-9239-125860130e89\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " Jul 02 18:50:20.119028 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.118805 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3569485e-42cc-47cd-9239-125860130e89-proxy-tls\") pod \"3569485e-42cc-47cd-9239-125860130e89\" (UID: \"3569485e-42cc-47cd-9239-125860130e89\") " Jul 02 18:50:20.119097 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.119074 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3569485e-42cc-47cd-9239-125860130e89-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3569485e-42cc-47cd-9239-125860130e89" (UID: "3569485e-42cc-47cd-9239-125860130e89"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:50:20.120827 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.120804 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3569485e-42cc-47cd-9239-125860130e89-kube-api-access-hxczp" (OuterVolumeSpecName: "kube-api-access-hxczp") pod "3569485e-42cc-47cd-9239-125860130e89" (UID: "3569485e-42cc-47cd-9239-125860130e89"). InnerVolumeSpecName "kube-api-access-hxczp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:50:20.120955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.120935 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3569485e-42cc-47cd-9239-125860130e89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3569485e-42cc-47cd-9239-125860130e89" (UID: "3569485e-42cc-47cd-9239-125860130e89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:50:20.219569 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.219538 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxczp\" (UniqueName: \"kubernetes.io/projected/3569485e-42cc-47cd-9239-125860130e89-kube-api-access-hxczp\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:50:20.219569 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.219563 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3569485e-42cc-47cd-9239-125860130e89-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:50:20.219569 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.219572 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3569485e-42cc-47cd-9239-125860130e89-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:50:20.478691 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.478630 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:50:20.825832 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.825747 2552 generic.go:356] "Generic (PLEG): container finished" podID="3569485e-42cc-47cd-9239-125860130e89" containerID="677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8" exitCode=0 Jul 02 18:50:20.825832 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.825801 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" Jul 02 18:50:20.825832 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.825814 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" event={"ID":"3569485e-42cc-47cd-9239-125860130e89","Type":"ContainerDied","Data":"677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8"} Jul 02 18:50:20.826069 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.825863 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6" event={"ID":"3569485e-42cc-47cd-9239-125860130e89","Type":"ContainerDied","Data":"1d8550989a7c29f6a4b0db20d37502ba5e04bf0da70e1f9729d430f024f9076e"} Jul 02 18:50:20.826069 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.825888 2552 scope.go:117] "RemoveContainer" containerID="cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f" Jul 02 18:50:20.835395 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.835378 2552 scope.go:117] "RemoveContainer" containerID="677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8" Jul 02 18:50:20.842790 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.842771 2552 scope.go:117] "RemoveContainer" containerID="cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f" Jul 02 18:50:20.843022 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:50:20.843004 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f\": container with ID starting with cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f not found: ID does not exist" containerID="cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f" Jul 02 18:50:20.843086 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.843035 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f"} err="failed to get container status \"cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f\": rpc error: code = NotFound desc = could not find container \"cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f\": container with ID starting with cff9f6c63ca677055aecdf17f83fc68508a945c1c5f177c3b6390fb071c5288f not found: ID does not exist" Jul 02 18:50:20.843086 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.843058 2552 scope.go:117] "RemoveContainer" containerID="677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8" Jul 02 18:50:20.843288 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:50:20.843271 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8\": container with ID starting with 677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8 not found: ID does not exist" containerID="677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8" Jul 02 18:50:20.843328 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.843295 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8"} err="failed to get container status \"677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8\": rpc error: code = NotFound desc = could not find container \"677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8\": container with ID starting with 677618c5fb8dfd763da488d815f2b5dc25053bbfb02529263fd03bff817c4fb8 not found: ID does not exist" Jul 02 18:50:20.847844 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.847823 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6"] Jul 02 18:50:20.852348 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:20.852328 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a5373-predictor-9c9cf6bb7-5kvh6"] Jul 02 18:50:21.236609 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:21.236579 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3569485e-42cc-47cd-9239-125860130e89" path="/var/lib/kubelet/pods/3569485e-42cc-47cd-9239-125860130e89/volumes" Jul 02 18:50:23.819614 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:23.819586 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:50:23.820195 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:23.820161 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 02 18:50:25.478501 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:25.478453 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:50:30.478564 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:30.478525 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:50:30.479045 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:30.478656 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:50:33.820559 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:33.820517 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 02 18:50:35.478315 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:35.478269 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:50:40.479338 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:40.479296 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:50:43.820249 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:43.820208 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 02 18:50:45.478806 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:45.478762 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:50:46.955971 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:46.955937 2552 generic.go:356] "Generic (PLEG): container finished" podID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerID="b7869031be38a902fe6adbfd70babc3337bba27a740e23cb7158aaeead7391a3" exitCode=0 Jul 02 18:50:46.956329 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:46.956013 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" event={"ID":"f4fd8493-acbe-4086-b17c-f43e0381c905","Type":"ContainerDied","Data":"b7869031be38a902fe6adbfd70babc3337bba27a740e23cb7158aaeead7391a3"} Jul 02 18:50:47.372417 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.372390 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:50:47.461609 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.461573 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls\") pod \"f4fd8493-acbe-4086-b17c-f43e0381c905\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " Jul 02 18:50:47.461770 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.461686 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fd8493-acbe-4086-b17c-f43e0381c905-openshift-service-ca-bundle\") pod \"f4fd8493-acbe-4086-b17c-f43e0381c905\" (UID: \"f4fd8493-acbe-4086-b17c-f43e0381c905\") " Jul 02 18:50:47.462083 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.462061 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fd8493-acbe-4086-b17c-f43e0381c905-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f4fd8493-acbe-4086-b17c-f43e0381c905" (UID: "f4fd8493-acbe-4086-b17c-f43e0381c905"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:50:47.463669 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.463625 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4fd8493-acbe-4086-b17c-f43e0381c905" (UID: "f4fd8493-acbe-4086-b17c-f43e0381c905"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:50:47.562590 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.562502 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4fd8493-acbe-4086-b17c-f43e0381c905-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:50:47.562590 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.562536 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fd8493-acbe-4086-b17c-f43e0381c905-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:50:47.961394 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.961356 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" event={"ID":"f4fd8493-acbe-4086-b17c-f43e0381c905","Type":"ContainerDied","Data":"b27241e97e2d357f44d2e7e18f2063a696464104e3fe5e6f0bdb735f81aa79f7"} Jul 02 18:50:47.961897 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.961425 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq" Jul 02 18:50:47.961897 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.961443 2552 scope.go:117] "RemoveContainer" containerID="b7869031be38a902fe6adbfd70babc3337bba27a740e23cb7158aaeead7391a3" Jul 02 18:50:47.983441 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.983415 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq"] Jul 02 18:50:47.987128 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:47.987105 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a5373-79d8b76794-sl4kq"] Jul 02 18:50:49.236509 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:49.236479 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" path="/var/lib/kubelet/pods/f4fd8493-acbe-4086-b17c-f43e0381c905/volumes" Jul 02 18:50:53.820494 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:53.820456 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 02 18:50:56.584343 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.584308 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n"] Jul 02 18:50:56.584875 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.584520 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" containerID="cri-o://893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b" gracePeriod=30 Jul 02 18:50:56.697900 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.697860 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d"] Jul 02 18:50:56.698213 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.698165 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" containerID="cri-o://0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c" gracePeriod=30 Jul 02 18:50:56.698892 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.698849 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kube-rbac-proxy" containerID="cri-o://ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb" gracePeriod=30 Jul 02 18:50:56.731012 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.730983 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs"] Jul 02 18:50:56.731376 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.731363 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" Jul 02 18:50:56.731419 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.731378 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" Jul 02 18:50:56.731419 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.731399 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kube-rbac-proxy" Jul 02 18:50:56.731419 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.731404 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kube-rbac-proxy" Jul 02 18:50:56.731419 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.731411 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" Jul 02 18:50:56.731419 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.731416 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" Jul 02 18:50:56.731861 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.731467 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4fd8493-acbe-4086-b17c-f43e0381c905" containerName="switch-graph-a5373" Jul 02 18:50:56.731861 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.731477 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kserve-container" Jul 02 18:50:56.731861 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.731484 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3569485e-42cc-47cd-9239-125860130e89" containerName="kube-rbac-proxy" Jul 02 18:50:56.735888 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.735870 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:56.737836 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.737813 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-94e8c-kube-rbac-proxy-sar-config\"" Jul 02 18:50:56.737926 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.737852 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-94e8c-predictor-serving-cert\"" Jul 02 18:50:56.742217 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.742197 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs"] Jul 02 18:50:56.847454 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.847370 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrck\" (UniqueName: \"kubernetes.io/projected/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-api-access-sdrck\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:56.848302 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.847506 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898521dc-c12d-49ba-91bb-d9bf1437cf21-proxy-tls\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:56.848302 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.847555 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:56.949114 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.949085 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrck\" (UniqueName: \"kubernetes.io/projected/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-api-access-sdrck\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:56.949314 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.949142 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898521dc-c12d-49ba-91bb-d9bf1437cf21-proxy-tls\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:56.949314 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.949257 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:56.949437 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:50:56.949372 2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-serving-cert: secret "success-200-isvc-94e8c-predictor-serving-cert" not found Jul 02 18:50:56.949437 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:50:56.949433 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898521dc-c12d-49ba-91bb-d9bf1437cf21-proxy-tls podName:898521dc-c12d-49ba-91bb-d9bf1437cf21 nodeName:}" failed. No retries permitted until 2026-07-02 18:50:57.44941363 +0000 UTC m=+1214.735763463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/898521dc-c12d-49ba-91bb-d9bf1437cf21-proxy-tls") pod "success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" (UID: "898521dc-c12d-49ba-91bb-d9bf1437cf21") : secret "success-200-isvc-94e8c-predictor-serving-cert" not found Jul 02 18:50:56.950064 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.950046 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:56.959460 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:56.959436 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrck\" (UniqueName: \"kubernetes.io/projected/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-api-access-sdrck\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:57.007783 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:57.007752 2552 generic.go:356] "Generic (PLEG): container finished" podID="d23f82c5-0e33-4821-a596-783a6938c9db" containerID="ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb" exitCode=2 Jul 02 18:50:57.007929 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:57.007828 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" event={"ID":"d23f82c5-0e33-4821-a596-783a6938c9db","Type":"ContainerDied","Data":"ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb"} Jul 02 18:50:57.452975 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:57.452924 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898521dc-c12d-49ba-91bb-d9bf1437cf21-proxy-tls\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:57.455436 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:57.455413 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898521dc-c12d-49ba-91bb-d9bf1437cf21-proxy-tls\") pod \"success-200-isvc-94e8c-predictor-654cd696d7-pjsrs\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:57.647111 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:57.647062 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:57.777392 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:57.777364 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs"] Jul 02 18:50:57.779498 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:50:57.779468 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898521dc_c12d_49ba_91bb_d9bf1437cf21.slice/crio-0f1cf0b3d54a7f158b8892354b7bedafec0c6cf386522521cdac0ecb1bcb7739 WatchSource:0}: Error finding container 0f1cf0b3d54a7f158b8892354b7bedafec0c6cf386522521cdac0ecb1bcb7739: Status 404 returned error can't find the container with id 0f1cf0b3d54a7f158b8892354b7bedafec0c6cf386522521cdac0ecb1bcb7739 Jul 02 18:50:58.013695 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:58.013596 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" event={"ID":"898521dc-c12d-49ba-91bb-d9bf1437cf21","Type":"ContainerStarted","Data":"4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae"} Jul 02 18:50:58.013695 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:58.013681 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" event={"ID":"898521dc-c12d-49ba-91bb-d9bf1437cf21","Type":"ContainerStarted","Data":"09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1"} Jul 02 18:50:58.013867 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:58.013714 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" event={"ID":"898521dc-c12d-49ba-91bb-d9bf1437cf21","Type":"ContainerStarted","Data":"0f1cf0b3d54a7f158b8892354b7bedafec0c6cf386522521cdac0ecb1bcb7739"} Jul 02 18:50:58.013904 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:58.013868 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:58.031870 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:58.031819 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podStartSLOduration=2.031804356 podStartE2EDuration="2.031804356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:50:58.029996402 +0000 UTC m=+1215.316346247" watchObservedRunningTime="2026-07-02 18:50:58.031804356 +0000 UTC m=+1215.318154199" Jul 02 18:50:59.019120 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.019085 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:50:59.020535 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.020506 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 02 18:50:59.419185 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.419096 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused" Jul 02 18:50:59.424782 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.424755 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 02 18:50:59.703783 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.703739 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:50:59.951622 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.951600 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:50:59.977188 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.977119 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23f82c5-0e33-4821-a596-783a6938c9db-kube-rbac-proxy-sar-config\") pod \"d23f82c5-0e33-4821-a596-783a6938c9db\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " Jul 02 18:50:59.977307 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.977196 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkgqc\" (UniqueName: \"kubernetes.io/projected/d23f82c5-0e33-4821-a596-783a6938c9db-kube-api-access-bkgqc\") pod \"d23f82c5-0e33-4821-a596-783a6938c9db\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " Jul 02 18:50:59.977307 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.977237 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23f82c5-0e33-4821-a596-783a6938c9db-proxy-tls\") pod \"d23f82c5-0e33-4821-a596-783a6938c9db\" (UID: \"d23f82c5-0e33-4821-a596-783a6938c9db\") " Jul 02 18:50:59.977520 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.977485 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23f82c5-0e33-4821-a596-783a6938c9db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d23f82c5-0e33-4821-a596-783a6938c9db" (UID: "d23f82c5-0e33-4821-a596-783a6938c9db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:50:59.979162 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.979136 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23f82c5-0e33-4821-a596-783a6938c9db-kube-api-access-bkgqc" (OuterVolumeSpecName: "kube-api-access-bkgqc") pod "d23f82c5-0e33-4821-a596-783a6938c9db" (UID: "d23f82c5-0e33-4821-a596-783a6938c9db"). InnerVolumeSpecName "kube-api-access-bkgqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:50:59.979249 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:50:59.979203 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23f82c5-0e33-4821-a596-783a6938c9db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d23f82c5-0e33-4821-a596-783a6938c9db" (UID: "d23f82c5-0e33-4821-a596-783a6938c9db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:51:00.024913 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.024878 2552 generic.go:356] "Generic (PLEG): container finished" podID="d23f82c5-0e33-4821-a596-783a6938c9db" containerID="0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c" exitCode=0 Jul 02 18:51:00.025318 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.024926 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" Jul 02 18:51:00.025318 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.024969 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" event={"ID":"d23f82c5-0e33-4821-a596-783a6938c9db","Type":"ContainerDied","Data":"0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c"} Jul 02 18:51:00.025318 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.025029 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d" event={"ID":"d23f82c5-0e33-4821-a596-783a6938c9db","Type":"ContainerDied","Data":"a3f545266743e5339f667d7e921bad31c241c6e741cc45ca734597e30b410bc9"} Jul 02 18:51:00.025318 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.025058 2552 scope.go:117] "RemoveContainer" containerID="ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb" Jul 02 18:51:00.025693 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.025656 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 02 18:51:00.033585 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.033568 2552 scope.go:117] "RemoveContainer" containerID="0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c" Jul 02 18:51:00.041052 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.041036 2552 scope.go:117] "RemoveContainer" containerID="ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb" Jul 02 18:51:00.041335 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:51:00.041316 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb\": container with ID starting with ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb not found: ID does not exist" containerID="ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb" Jul 02 18:51:00.041400 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.041341 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb"} err="failed to get container status \"ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb\": rpc error: code = NotFound desc = could not find container \"ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb\": container with ID starting with ee75d9a353160f8c6a1ec971947875051c6a66ca9afe7b665e159aa4436f20cb not found: ID does not exist" Jul 02 18:51:00.041400 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.041357 2552 scope.go:117] "RemoveContainer" containerID="0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c" Jul 02 18:51:00.041656 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:51:00.041586 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c\": container with ID starting with 0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c not found: ID does not exist" containerID="0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c" Jul 02 18:51:00.041788 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.041684 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c"} err="failed to get container status \"0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c\": rpc error: code = NotFound desc = could not find container \"0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c\": container with ID starting with 0d5ba799c88bc89fdbbee892c50ae5d2359f15af6a296ef5745682da280d3b5c not found: ID does not exist" Jul 02 18:51:00.044765 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.044744 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d"] Jul 02 18:51:00.046331 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.046313 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8b76-predictor-66f687f9b9-j5g9d"] Jul 02 18:51:00.078090 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.078064 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23f82c5-0e33-4821-a596-783a6938c9db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:00.078090 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.078088 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkgqc\" (UniqueName: \"kubernetes.io/projected/d23f82c5-0e33-4821-a596-783a6938c9db-kube-api-access-bkgqc\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:00.078305 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:00.078100 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23f82c5-0e33-4821-a596-783a6938c9db-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:01.236286 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:01.236253 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" path="/var/lib/kubelet/pods/d23f82c5-0e33-4821-a596-783a6938c9db/volumes" Jul 02 18:51:03.820785 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:03.820756 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:51:04.703348 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:04.703310 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:05.029401 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:05.029320 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:51:05.029799 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:05.029774 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 02 18:51:09.704324 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:09.704281 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:09.704732 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:09.704398 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:51:14.704025 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:14.703980 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:15.030184 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:15.030094 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 02 18:51:16.932347 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.932314 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z"] Jul 02 18:51:16.932916 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.932897 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" Jul 02 18:51:16.933004 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.932918 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" Jul 02 18:51:16.933004 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.932940 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kube-rbac-proxy" Jul 02 18:51:16.933004 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.932948 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kube-rbac-proxy" Jul 02 18:51:16.933160 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.933037 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kube-rbac-proxy" Jul 02 18:51:16.933160 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.933052 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23f82c5-0e33-4821-a596-783a6938c9db" containerName="kserve-container" Jul 02 18:51:16.935910 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.935889 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:16.937809 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.937789 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e22cf-serving-cert\"" Jul 02 18:51:16.937921 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.937791 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e22cf-kube-rbac-proxy-sar-config\"" Jul 02 18:51:16.945687 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:16.945668 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z"] Jul 02 18:51:17.027703 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:17.027625 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/196bedf0-44de-4568-8804-6506181b7e5f-openshift-service-ca-bundle\") pod \"ensemble-graph-e22cf-5c9b96dc46-s9n6z\" (UID: \"196bedf0-44de-4568-8804-6506181b7e5f\") " pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:17.027858 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:17.027775 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196bedf0-44de-4568-8804-6506181b7e5f-proxy-tls\") pod \"ensemble-graph-e22cf-5c9b96dc46-s9n6z\" (UID: \"196bedf0-44de-4568-8804-6506181b7e5f\") " pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:17.129186 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:17.129151 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/196bedf0-44de-4568-8804-6506181b7e5f-openshift-service-ca-bundle\") pod \"ensemble-graph-e22cf-5c9b96dc46-s9n6z\" (UID: \"196bedf0-44de-4568-8804-6506181b7e5f\") " pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:17.129343 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:17.129225 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196bedf0-44de-4568-8804-6506181b7e5f-proxy-tls\") pod \"ensemble-graph-e22cf-5c9b96dc46-s9n6z\" (UID: \"196bedf0-44de-4568-8804-6506181b7e5f\") " pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:17.129343 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:51:17.129325 2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-e22cf-serving-cert: secret "ensemble-graph-e22cf-serving-cert" not found Jul 02 18:51:17.129417 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:51:17.129389 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/196bedf0-44de-4568-8804-6506181b7e5f-proxy-tls podName:196bedf0-44de-4568-8804-6506181b7e5f nodeName:}" failed. No retries permitted until 2026-07-02 18:51:17.629372689 +0000 UTC m=+1234.915722509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/196bedf0-44de-4568-8804-6506181b7e5f-proxy-tls") pod "ensemble-graph-e22cf-5c9b96dc46-s9n6z" (UID: "196bedf0-44de-4568-8804-6506181b7e5f") : secret "ensemble-graph-e22cf-serving-cert" not found Jul 02 18:51:17.129786 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:17.129767 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/196bedf0-44de-4568-8804-6506181b7e5f-openshift-service-ca-bundle\") pod \"ensemble-graph-e22cf-5c9b96dc46-s9n6z\" (UID: \"196bedf0-44de-4568-8804-6506181b7e5f\") " pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:17.632882 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:17.632823 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196bedf0-44de-4568-8804-6506181b7e5f-proxy-tls\") pod \"ensemble-graph-e22cf-5c9b96dc46-s9n6z\" (UID: \"196bedf0-44de-4568-8804-6506181b7e5f\") " pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:17.635239 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:17.635215 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196bedf0-44de-4568-8804-6506181b7e5f-proxy-tls\") pod \"ensemble-graph-e22cf-5c9b96dc46-s9n6z\" (UID: \"196bedf0-44de-4568-8804-6506181b7e5f\") " pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:17.846605 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:17.846568 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:17.974019 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:17.973995 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z"] Jul 02 18:51:17.976065 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:51:17.976039 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196bedf0_44de_4568_8804_6506181b7e5f.slice/crio-1005c5056e925b325473dc55995ce743096e54ba1f1c6ecd0f25cda6f4392e1e WatchSource:0}: Error finding container 1005c5056e925b325473dc55995ce743096e54ba1f1c6ecd0f25cda6f4392e1e: Status 404 returned error can't find the container with id 1005c5056e925b325473dc55995ce743096e54ba1f1c6ecd0f25cda6f4392e1e Jul 02 18:51:18.110481 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:18.110428 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" event={"ID":"196bedf0-44de-4568-8804-6506181b7e5f","Type":"ContainerStarted","Data":"9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7"} Jul 02 18:51:18.110766 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:18.110592 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:18.110766 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:18.110623 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" event={"ID":"196bedf0-44de-4568-8804-6506181b7e5f","Type":"ContainerStarted","Data":"1005c5056e925b325473dc55995ce743096e54ba1f1c6ecd0f25cda6f4392e1e"} Jul 02 18:51:18.125992 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:18.125952 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" podStartSLOduration=2.125938473 podStartE2EDuration="2.125938473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:51:18.124813942 +0000 UTC m=+1235.411163786" watchObservedRunningTime="2026-07-02 18:51:18.125938473 +0000 UTC m=+1235.412288316" Jul 02 18:51:19.703676 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:19.703608 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:24.121213 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:24.121177 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:24.703380 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:24.703336 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:25.030678 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:25.030569 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 02 18:51:26.740687 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:26.740666 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:51:26.811653 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:26.811603 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7715988c-7201-4301-9699-e351641dafbf-proxy-tls\") pod \"7715988c-7201-4301-9699-e351641dafbf\" (UID: \"7715988c-7201-4301-9699-e351641dafbf\") " Jul 02 18:51:26.811824 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:26.811669 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7715988c-7201-4301-9699-e351641dafbf-openshift-service-ca-bundle\") pod \"7715988c-7201-4301-9699-e351641dafbf\" (UID: \"7715988c-7201-4301-9699-e351641dafbf\") " Jul 02 18:51:26.812014 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:26.811990 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7715988c-7201-4301-9699-e351641dafbf-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7715988c-7201-4301-9699-e351641dafbf" (UID: "7715988c-7201-4301-9699-e351641dafbf"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:51:26.813673 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:26.813649 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7715988c-7201-4301-9699-e351641dafbf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7715988c-7201-4301-9699-e351641dafbf" (UID: "7715988c-7201-4301-9699-e351641dafbf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:51:26.912604 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:26.912514 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7715988c-7201-4301-9699-e351641dafbf-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:26.912604 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:26.912545 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7715988c-7201-4301-9699-e351641dafbf-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:27.007939 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.007907 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z"] Jul 02 18:51:27.008144 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.008122 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" containerID="cri-o://9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7" gracePeriod=30 Jul 02 18:51:27.132663 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.132613 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv"] Jul 02 18:51:27.132955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.132926 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" containerID="cri-o://c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76" gracePeriod=30 Jul 02 18:51:27.133062 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.132972 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kube-rbac-proxy" containerID="cri-o://c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4" gracePeriod=30 Jul 02 18:51:27.153936 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.153908 2552 generic.go:356] "Generic (PLEG): container finished" podID="7715988c-7201-4301-9699-e351641dafbf" containerID="893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b" exitCode=0 Jul 02 18:51:27.154036 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.153965 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" Jul 02 18:51:27.154036 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.153991 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" event={"ID":"7715988c-7201-4301-9699-e351641dafbf","Type":"ContainerDied","Data":"893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b"} Jul 02 18:51:27.154107 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.154042 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n" event={"ID":"7715988c-7201-4301-9699-e351641dafbf","Type":"ContainerDied","Data":"0053705c55a613f9cfc9f26dee53df3a22b0b741727d0b13950021e1782a9e03"} Jul 02 18:51:27.154107 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.154065 2552 scope.go:117] "RemoveContainer" containerID="893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b" Jul 02 18:51:27.163041 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.162990 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf"] Jul 02 18:51:27.163412 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.163399 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" Jul 02 18:51:27.163464 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.163415 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" Jul 02 18:51:27.163499 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.163473 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7715988c-7201-4301-9699-e351641dafbf" containerName="sequence-graph-e8b76" Jul 02 18:51:27.166908 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.166891 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.168447 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.168422 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7acc8-predictor-serving-cert\"" Jul 02 18:51:27.168805 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.168787 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7acc8-kube-rbac-proxy-sar-config\"" Jul 02 18:51:27.173944 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.173918 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf"] Jul 02 18:51:27.178620 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.178600 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n"] Jul 02 18:51:27.186486 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.186466 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8b76-cc658d568-n4p7n"] Jul 02 18:51:27.238668 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.238617 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7715988c-7201-4301-9699-e351641dafbf" path="/var/lib/kubelet/pods/7715988c-7201-4301-9699-e351641dafbf/volumes" Jul 02 18:51:27.247609 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.247590 2552 scope.go:117] "RemoveContainer" containerID="893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b" Jul 02 18:51:27.247998 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:51:27.247975 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b\": container with ID starting with 893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b not found: ID does not exist" containerID="893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b" Jul 02 18:51:27.248088 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.248010 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b"} err="failed to get container status \"893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b\": rpc error: code = NotFound desc = could not find container \"893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b\": container with ID starting with 893b78dc389ca297530e6e497e20cf2ded5f2196e8164aa93d021560d1c3a15b not found: ID does not exist" Jul 02 18:51:27.315630 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.315587 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-proxy-tls\") pod \"success-200-isvc-7acc8-predictor-5896b7c7c-74fgf\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.315842 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.315733 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79sbt\" (UniqueName: \"kubernetes.io/projected/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-api-access-79sbt\") pod \"success-200-isvc-7acc8-predictor-5896b7c7c-74fgf\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.315896 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.315869 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7acc8-predictor-5896b7c7c-74fgf\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.417035 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.416949 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-proxy-tls\") pod \"success-200-isvc-7acc8-predictor-5896b7c7c-74fgf\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.417035 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.416997 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79sbt\" (UniqueName: \"kubernetes.io/projected/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-api-access-79sbt\") pod \"success-200-isvc-7acc8-predictor-5896b7c7c-74fgf\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.417244 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.417036 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7acc8-predictor-5896b7c7c-74fgf\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.417729 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.417709 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7acc8-predictor-5896b7c7c-74fgf\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.419405 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.419386 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-proxy-tls\") pod \"success-200-isvc-7acc8-predictor-5896b7c7c-74fgf\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.424670 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.424627 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79sbt\" (UniqueName: \"kubernetes.io/projected/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-api-access-79sbt\") pod \"success-200-isvc-7acc8-predictor-5896b7c7c-74fgf\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.478175 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.478141 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:27.617244 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:27.616986 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf"] Jul 02 18:51:27.619604 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:51:27.619565 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffc5bbc_f8c3_4ba0_839c_d20e01d598bb.slice/crio-cb46905146c0f1fe626e6dc72a7e5701f3c98765055800fe78bce8de9fe6beda WatchSource:0}: Error finding container cb46905146c0f1fe626e6dc72a7e5701f3c98765055800fe78bce8de9fe6beda: Status 404 returned error can't find the container with id cb46905146c0f1fe626e6dc72a7e5701f3c98765055800fe78bce8de9fe6beda Jul 02 18:51:28.161047 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:28.161007 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" event={"ID":"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb","Type":"ContainerStarted","Data":"0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861"} Jul 02 18:51:28.161482 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:28.161078 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" event={"ID":"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb","Type":"ContainerStarted","Data":"a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e"} Jul 02 18:51:28.161482 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:28.161113 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" event={"ID":"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb","Type":"ContainerStarted","Data":"cb46905146c0f1fe626e6dc72a7e5701f3c98765055800fe78bce8de9fe6beda"} Jul 02 18:51:28.161482 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:28.161147 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:28.164723 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:28.164693 2552 generic.go:356] "Generic (PLEG): container finished" podID="104b64ce-542f-4113-9fea-d346c4f4991e" containerID="c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4" exitCode=2 Jul 02 18:51:28.164908 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:28.164764 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" event={"ID":"104b64ce-542f-4113-9fea-d346c4f4991e","Type":"ContainerDied","Data":"c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4"} Jul 02 18:51:28.176518 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:28.176462 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podStartSLOduration=1.176443612 podStartE2EDuration="1.176443612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:51:28.175613621 +0000 UTC m=+1245.461963477" watchObservedRunningTime="2026-07-02 18:51:28.176443612 +0000 UTC m=+1245.462793455" Jul 02 18:51:28.816039 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:28.815997 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused" Jul 02 18:51:29.119051 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:29.118965 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:29.171706 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:29.171624 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:29.173029 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:29.173001 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 02 18:51:30.177261 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.177233 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 02 18:51:30.395748 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.395721 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:51:30.544698 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.544664 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b64ce-542f-4113-9fea-d346c4f4991e-kube-rbac-proxy-sar-config\") pod \"104b64ce-542f-4113-9fea-d346c4f4991e\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " Jul 02 18:51:30.544848 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.544735 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcnwt\" (UniqueName: \"kubernetes.io/projected/104b64ce-542f-4113-9fea-d346c4f4991e-kube-api-access-tcnwt\") pod \"104b64ce-542f-4113-9fea-d346c4f4991e\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " Jul 02 18:51:30.544848 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.544834 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b64ce-542f-4113-9fea-d346c4f4991e-proxy-tls\") pod \"104b64ce-542f-4113-9fea-d346c4f4991e\" (UID: \"104b64ce-542f-4113-9fea-d346c4f4991e\") " Jul 02 18:51:30.545049 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.545024 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104b64ce-542f-4113-9fea-d346c4f4991e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "104b64ce-542f-4113-9fea-d346c4f4991e" (UID: "104b64ce-542f-4113-9fea-d346c4f4991e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:51:30.545199 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.545107 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104b64ce-542f-4113-9fea-d346c4f4991e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:30.546847 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.546825 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104b64ce-542f-4113-9fea-d346c4f4991e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "104b64ce-542f-4113-9fea-d346c4f4991e" (UID: "104b64ce-542f-4113-9fea-d346c4f4991e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:51:30.546910 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.546880 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104b64ce-542f-4113-9fea-d346c4f4991e-kube-api-access-tcnwt" (OuterVolumeSpecName: "kube-api-access-tcnwt") pod "104b64ce-542f-4113-9fea-d346c4f4991e" (UID: "104b64ce-542f-4113-9fea-d346c4f4991e"). InnerVolumeSpecName "kube-api-access-tcnwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:51:30.645973 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.645934 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcnwt\" (UniqueName: \"kubernetes.io/projected/104b64ce-542f-4113-9fea-d346c4f4991e-kube-api-access-tcnwt\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:30.645973 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:30.645966 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104b64ce-542f-4113-9fea-d346c4f4991e-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:31.182959 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.182928 2552 generic.go:356] "Generic (PLEG): container finished" podID="104b64ce-542f-4113-9fea-d346c4f4991e" containerID="c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76" exitCode=0 Jul 02 18:51:31.183328 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.182995 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" Jul 02 18:51:31.183328 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.183012 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" event={"ID":"104b64ce-542f-4113-9fea-d346c4f4991e","Type":"ContainerDied","Data":"c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76"} Jul 02 18:51:31.183328 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.183092 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv" event={"ID":"104b64ce-542f-4113-9fea-d346c4f4991e","Type":"ContainerDied","Data":"98f5d2dfd285b0e6514c411adc576d0466541a292bd2747524baf46b8bc0cc83"} Jul 02 18:51:31.183328 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.183143 2552 scope.go:117] "RemoveContainer" containerID="c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4" Jul 02 18:51:31.192445 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.192426 2552 scope.go:117] "RemoveContainer" containerID="c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76" Jul 02 18:51:31.200068 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.200049 2552 scope.go:117] "RemoveContainer" containerID="c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4" Jul 02 18:51:31.200325 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:51:31.200305 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4\": container with ID starting with c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4 not found: ID does not exist" containerID="c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4" Jul 02 18:51:31.200392 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.200338 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4"} err="failed to get container status \"c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4\": rpc error: code = NotFound desc = could not find container \"c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4\": container with ID starting with c3744f6b1756a82b1819b3546927914c998599b5925eacd9ccc2db627ef449d4 not found: ID does not exist" Jul 02 18:51:31.200392 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.200380 2552 scope.go:117] "RemoveContainer" containerID="c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76" Jul 02 18:51:31.200658 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:51:31.200622 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76\": container with ID starting with c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76 not found: ID does not exist" containerID="c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76" Jul 02 18:51:31.200745 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.200663 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76"} err="failed to get container status \"c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76\": rpc error: code = NotFound desc = could not find container \"c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76\": container with ID starting with c0b8b57eb4316d1b7fbdaba167de9a43d75285d4c9ae83ca60ef10e2cd67cd76 not found: ID does not exist" Jul 02 18:51:31.203414 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.203394 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv"] Jul 02 18:51:31.209896 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.209875 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e22cf-predictor-7dd874c4cc-fxtqv"] Jul 02 18:51:31.236855 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:31.236835 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" path="/var/lib/kubelet/pods/104b64ce-542f-4113-9fea-d346c4f4991e/volumes" Jul 02 18:51:34.119797 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:34.119761 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:35.029835 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:35.029793 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 02 18:51:35.182403 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:35.182370 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:51:35.182877 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:35.182838 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 02 18:51:39.119937 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:39.119897 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:39.120300 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:39.120007 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:44.119995 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:44.119955 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:45.030497 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:45.030465 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:51:45.182923 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:45.182882 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 02 18:51:49.119203 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:49.119156 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:54.119864 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:54.119826 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:51:55.182988 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:55.182947 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 02 18:51:56.752955 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.752926 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54"] Jul 02 18:51:56.753358 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.753345 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" Jul 02 18:51:56.753404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.753360 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" Jul 02 18:51:56.753404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.753380 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kube-rbac-proxy" Jul 02 18:51:56.753404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.753389 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kube-rbac-proxy" Jul 02 18:51:56.753494 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.753444 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kube-rbac-proxy" Jul 02 18:51:56.753494 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.753457 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="104b64ce-542f-4113-9fea-d346c4f4991e" containerName="kserve-container" Jul 02 18:51:56.758259 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.758244 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:51:56.760246 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.760218 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-94e8c-kube-rbac-proxy-sar-config\"" Jul 02 18:51:56.760390 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.760316 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-94e8c-serving-cert\"" Jul 02 18:51:56.765038 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.765017 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54"] Jul 02 18:51:56.869659 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.869584 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d06625-9504-4963-a5c6-656c3fc8fb94-proxy-tls\") pod \"sequence-graph-94e8c-5775877b49-zkm54\" (UID: \"d3d06625-9504-4963-a5c6-656c3fc8fb94\") " pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:51:56.869879 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.869758 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3d06625-9504-4963-a5c6-656c3fc8fb94-openshift-service-ca-bundle\") pod \"sequence-graph-94e8c-5775877b49-zkm54\" (UID: \"d3d06625-9504-4963-a5c6-656c3fc8fb94\") " pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:51:56.971048 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.971009 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d06625-9504-4963-a5c6-656c3fc8fb94-proxy-tls\") pod \"sequence-graph-94e8c-5775877b49-zkm54\" (UID: \"d3d06625-9504-4963-a5c6-656c3fc8fb94\") " pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:51:56.971247 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.971173 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3d06625-9504-4963-a5c6-656c3fc8fb94-openshift-service-ca-bundle\") pod \"sequence-graph-94e8c-5775877b49-zkm54\" (UID: \"d3d06625-9504-4963-a5c6-656c3fc8fb94\") " pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:51:56.971837 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.971816 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3d06625-9504-4963-a5c6-656c3fc8fb94-openshift-service-ca-bundle\") pod \"sequence-graph-94e8c-5775877b49-zkm54\" (UID: \"d3d06625-9504-4963-a5c6-656c3fc8fb94\") " pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:51:56.973415 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:56.973394 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d06625-9504-4963-a5c6-656c3fc8fb94-proxy-tls\") pod \"sequence-graph-94e8c-5775877b49-zkm54\" (UID: \"d3d06625-9504-4963-a5c6-656c3fc8fb94\") " pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:51:57.069048 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.069020 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:51:57.200310 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.200282 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:57.208297 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.208277 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54"] Jul 02 18:51:57.210407 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:51:57.210381 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d06625_9504_4963_a5c6_656c3fc8fb94.slice/crio-76b647fecbed351b83a4882af4b1749753bad2c97858bfcba25e423efc911a6a WatchSource:0}: Error finding container 76b647fecbed351b83a4882af4b1749753bad2c97858bfcba25e423efc911a6a: Status 404 returned error can't find the container with id 76b647fecbed351b83a4882af4b1749753bad2c97858bfcba25e423efc911a6a Jul 02 18:51:57.273127 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.273060 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196bedf0-44de-4568-8804-6506181b7e5f-proxy-tls\") pod \"196bedf0-44de-4568-8804-6506181b7e5f\" (UID: \"196bedf0-44de-4568-8804-6506181b7e5f\") " Jul 02 18:51:57.273252 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.273166 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/196bedf0-44de-4568-8804-6506181b7e5f-openshift-service-ca-bundle\") pod \"196bedf0-44de-4568-8804-6506181b7e5f\" (UID: \"196bedf0-44de-4568-8804-6506181b7e5f\") " Jul 02 18:51:57.273554 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.273524 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196bedf0-44de-4568-8804-6506181b7e5f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "196bedf0-44de-4568-8804-6506181b7e5f" (UID: "196bedf0-44de-4568-8804-6506181b7e5f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:51:57.275017 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.274994 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196bedf0-44de-4568-8804-6506181b7e5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "196bedf0-44de-4568-8804-6506181b7e5f" (UID: "196bedf0-44de-4568-8804-6506181b7e5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:51:57.309261 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.309222 2552 generic.go:356] "Generic (PLEG): container finished" podID="196bedf0-44de-4568-8804-6506181b7e5f" containerID="9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7" exitCode=0 Jul 02 18:51:57.309398 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.309265 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" Jul 02 18:51:57.309398 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.309282 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" event={"ID":"196bedf0-44de-4568-8804-6506181b7e5f","Type":"ContainerDied","Data":"9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7"} Jul 02 18:51:57.309398 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.309330 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z" event={"ID":"196bedf0-44de-4568-8804-6506181b7e5f","Type":"ContainerDied","Data":"1005c5056e925b325473dc55995ce743096e54ba1f1c6ecd0f25cda6f4392e1e"} Jul 02 18:51:57.309398 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.309349 2552 scope.go:117] "RemoveContainer" containerID="9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7" Jul 02 18:51:57.310944 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.310921 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" event={"ID":"d3d06625-9504-4963-a5c6-656c3fc8fb94","Type":"ContainerStarted","Data":"5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195"} Jul 02 18:51:57.311053 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.310962 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" event={"ID":"d3d06625-9504-4963-a5c6-656c3fc8fb94","Type":"ContainerStarted","Data":"76b647fecbed351b83a4882af4b1749753bad2c97858bfcba25e423efc911a6a"} Jul 02 18:51:57.311099 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.311055 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:51:57.318380 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.318365 2552 scope.go:117] "RemoveContainer" containerID="9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7" Jul 02 18:51:57.318663 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:51:57.318629 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7\": container with ID starting with 9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7 not found: ID does not exist" containerID="9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7" Jul 02 18:51:57.318713 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.318672 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7"} err="failed to get container status \"9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7\": rpc error: code = NotFound desc = could not find container \"9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7\": container with ID starting with 9a866d3c1e7ef7d3facf420003c1e72cd2dc376248559bedec3572c41b4652d7 not found: ID does not exist" Jul 02 18:51:57.327317 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.327279 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" podStartSLOduration=1.327267284 podStartE2EDuration="1.327267284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:51:57.325551267 +0000 UTC m=+1274.611901112" watchObservedRunningTime="2026-07-02 18:51:57.327267284 +0000 UTC m=+1274.613617126" Jul 02 18:51:57.339735 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.339713 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z"] Jul 02 18:51:57.346735 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.346710 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e22cf-5c9b96dc46-s9n6z"] Jul 02 18:51:57.373972 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.373938 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/196bedf0-44de-4568-8804-6506181b7e5f-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:57.373972 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:57.373970 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196bedf0-44de-4568-8804-6506181b7e5f-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:51:59.236735 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:51:59.236696 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="196bedf0-44de-4568-8804-6506181b7e5f" path="/var/lib/kubelet/pods/196bedf0-44de-4568-8804-6506181b7e5f/volumes" Jul 02 18:52:03.322404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:03.322373 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:52:05.183383 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:05.183346 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 02 18:52:06.843593 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.843558 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54"] Jul 02 18:52:06.844106 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.843784 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" containerID="cri-o://5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195" gracePeriod=30 Jul 02 18:52:06.966858 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.966820 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs"] Jul 02 18:52:06.967153 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.967118 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" containerID="cri-o://09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1" gracePeriod=30 Jul 02 18:52:06.967306 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.967151 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kube-rbac-proxy" containerID="cri-o://4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae" gracePeriod=30 Jul 02 18:52:06.990744 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.990713 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz"] Jul 02 18:52:06.991100 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.991088 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" Jul 02 18:52:06.991141 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.991101 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" Jul 02 18:52:06.991179 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.991172 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="196bedf0-44de-4568-8804-6506181b7e5f" containerName="ensemble-graph-e22cf" Jul 02 18:52:06.995885 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.995864 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:06.997940 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.997916 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ae135-predictor-serving-cert\"" Jul 02 18:52:06.998058 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:06.997969 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ae135-kube-rbac-proxy-sar-config\"" Jul 02 18:52:07.001405 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.001375 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz"] Jul 02 18:52:07.063449 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.063417 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pccf\" (UniqueName: \"kubernetes.io/projected/533fd46b-98d4-4392-975d-79511afd23fb-kube-api-access-8pccf\") pod \"success-200-isvc-ae135-predictor-686c9c7795-6p9jz\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.063664 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.063474 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533fd46b-98d4-4392-975d-79511afd23fb-proxy-tls\") pod \"success-200-isvc-ae135-predictor-686c9c7795-6p9jz\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.063664 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.063500 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/533fd46b-98d4-4392-975d-79511afd23fb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ae135-predictor-686c9c7795-6p9jz\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.165084 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.164977 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pccf\" (UniqueName: \"kubernetes.io/projected/533fd46b-98d4-4392-975d-79511afd23fb-kube-api-access-8pccf\") pod \"success-200-isvc-ae135-predictor-686c9c7795-6p9jz\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.165084 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.165042 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533fd46b-98d4-4392-975d-79511afd23fb-proxy-tls\") pod \"success-200-isvc-ae135-predictor-686c9c7795-6p9jz\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.165084 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.165066 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/533fd46b-98d4-4392-975d-79511afd23fb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ae135-predictor-686c9c7795-6p9jz\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.165916 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.165878 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/533fd46b-98d4-4392-975d-79511afd23fb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ae135-predictor-686c9c7795-6p9jz\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.168050 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.168024 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533fd46b-98d4-4392-975d-79511afd23fb-proxy-tls\") pod \"success-200-isvc-ae135-predictor-686c9c7795-6p9jz\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.172854 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.172830 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pccf\" (UniqueName: \"kubernetes.io/projected/533fd46b-98d4-4392-975d-79511afd23fb-kube-api-access-8pccf\") pod \"success-200-isvc-ae135-predictor-686c9c7795-6p9jz\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.308335 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.308297 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:07.363096 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.362968 2552 generic.go:356] "Generic (PLEG): container finished" podID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerID="4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae" exitCode=2 Jul 02 18:52:07.363096 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.363019 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" event={"ID":"898521dc-c12d-49ba-91bb-d9bf1437cf21","Type":"ContainerDied","Data":"4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae"} Jul 02 18:52:07.440120 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:07.440054 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz"] Jul 02 18:52:07.442329 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:52:07.442306 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533fd46b_98d4_4392_975d_79511afd23fb.slice/crio-eba9b1d7bf718a5e2861fd78f5d3b2fecf7e07a62d7b8e86cb359be506cea10a WatchSource:0}: Error finding container eba9b1d7bf718a5e2861fd78f5d3b2fecf7e07a62d7b8e86cb359be506cea10a: Status 404 returned error can't find the container with id eba9b1d7bf718a5e2861fd78f5d3b2fecf7e07a62d7b8e86cb359be506cea10a Jul 02 18:52:08.321291 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:08.321249 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:52:08.371594 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:08.371553 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" event={"ID":"533fd46b-98d4-4392-975d-79511afd23fb","Type":"ContainerStarted","Data":"289dcb6420212f769c4d46f61353e233973c402945f57c06f914161c91dd79b4"} Jul 02 18:52:08.371788 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:08.371761 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:08.371837 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:08.371806 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" event={"ID":"533fd46b-98d4-4392-975d-79511afd23fb","Type":"ContainerStarted","Data":"9551617cda14d58fb9f115ebc280c247d2f8d9b2dc148bc4da778ad37d2cd706"} Jul 02 18:52:08.371837 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:08.371829 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" event={"ID":"533fd46b-98d4-4392-975d-79511afd23fb","Type":"ContainerStarted","Data":"eba9b1d7bf718a5e2861fd78f5d3b2fecf7e07a62d7b8e86cb359be506cea10a"} Jul 02 18:52:08.391347 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:08.391294 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" podStartSLOduration=2.391278075 podStartE2EDuration="2.391278075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:52:08.388813024 +0000 UTC m=+1285.675162893" watchObservedRunningTime="2026-07-02 18:52:08.391278075 +0000 UTC m=+1285.677627919" Jul 02 18:52:09.376885 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:09.376855 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:09.378189 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:09.378163 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 02 18:52:10.025797 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.025754 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused" Jul 02 18:52:10.232337 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.232313 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:52:10.291918 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.291831 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdrck\" (UniqueName: \"kubernetes.io/projected/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-api-access-sdrck\") pod \"898521dc-c12d-49ba-91bb-d9bf1437cf21\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " Jul 02 18:52:10.291918 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.291876 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898521dc-c12d-49ba-91bb-d9bf1437cf21-proxy-tls\") pod \"898521dc-c12d-49ba-91bb-d9bf1437cf21\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " Jul 02 18:52:10.291918 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.291917 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-rbac-proxy-sar-config\") pod \"898521dc-c12d-49ba-91bb-d9bf1437cf21\" (UID: \"898521dc-c12d-49ba-91bb-d9bf1437cf21\") " Jul 02 18:52:10.292299 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.292276 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "898521dc-c12d-49ba-91bb-d9bf1437cf21" (UID: "898521dc-c12d-49ba-91bb-d9bf1437cf21"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:52:10.294065 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.294043 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-api-access-sdrck" (OuterVolumeSpecName: "kube-api-access-sdrck") pod "898521dc-c12d-49ba-91bb-d9bf1437cf21" (UID: "898521dc-c12d-49ba-91bb-d9bf1437cf21"). InnerVolumeSpecName "kube-api-access-sdrck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 18:52:10.294151 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.294085 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898521dc-c12d-49ba-91bb-d9bf1437cf21-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "898521dc-c12d-49ba-91bb-d9bf1437cf21" (UID: "898521dc-c12d-49ba-91bb-d9bf1437cf21"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:52:10.382776 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.382745 2552 generic.go:356] "Generic (PLEG): container finished" podID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerID="09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1" exitCode=0 Jul 02 18:52:10.383186 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.382792 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" Jul 02 18:52:10.383186 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.382828 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" event={"ID":"898521dc-c12d-49ba-91bb-d9bf1437cf21","Type":"ContainerDied","Data":"09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1"} Jul 02 18:52:10.383186 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.382883 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs" event={"ID":"898521dc-c12d-49ba-91bb-d9bf1437cf21","Type":"ContainerDied","Data":"0f1cf0b3d54a7f158b8892354b7bedafec0c6cf386522521cdac0ecb1bcb7739"} Jul 02 18:52:10.383186 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.382912 2552 scope.go:117] "RemoveContainer" containerID="4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae" Jul 02 18:52:10.383404 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.383380 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 02 18:52:10.392103 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.392086 2552 scope.go:117] "RemoveContainer" containerID="09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1" Jul 02 18:52:10.392420 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.392402 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrck\" (UniqueName: \"kubernetes.io/projected/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-api-access-sdrck\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:52:10.392499 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.392421 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898521dc-c12d-49ba-91bb-d9bf1437cf21-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:52:10.392499 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.392433 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898521dc-c12d-49ba-91bb-d9bf1437cf21-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:52:10.399390 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.399374 2552 scope.go:117] "RemoveContainer" containerID="4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae" Jul 02 18:52:10.399623 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:52:10.399603 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae\": container with ID starting with 4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae not found: ID does not exist" containerID="4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae" Jul 02 18:52:10.399867 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.399673 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae"} err="failed to get container status \"4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae\": rpc error: code = NotFound desc = could not find container \"4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae\": container with ID starting with 4ac1a58b6bdbf6b800f11f0c6cd84b2ce3720c6108bb087c7ba042b022eea1ae not found: ID does not exist" Jul 02 18:52:10.399867 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.399699 2552 scope.go:117] "RemoveContainer" containerID="09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1" Jul 02 18:52:10.400043 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:52:10.400024 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1\": container with ID starting with 09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1 not found: ID does not exist" containerID="09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1" Jul 02 18:52:10.400089 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.400051 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1"} err="failed to get container status \"09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1\": rpc error: code = NotFound desc = could not find container \"09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1\": container with ID starting with 09ad42230a41d7658d528be684a313fb02df6a74e1589b5f702a33605f4163f1 not found: ID does not exist" Jul 02 18:52:10.402421 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.402402 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs"] Jul 02 18:52:10.406136 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:10.406116 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94e8c-predictor-654cd696d7-pjsrs"] Jul 02 18:52:11.236711 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:11.236684 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" path="/var/lib/kubelet/pods/898521dc-c12d-49ba-91bb-d9bf1437cf21/volumes" Jul 02 18:52:13.320683 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:13.320627 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:52:15.184055 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:15.184024 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 18:52:15.387779 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:15.387752 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:52:15.388223 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:15.388195 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 02 18:52:18.320790 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:18.320749 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:52:18.321167 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:18.320854 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:52:23.321497 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:23.321457 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:52:25.388308 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:25.388268 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 02 18:52:27.237661 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.237603 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm"] Jul 02 18:52:27.238180 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.238160 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kube-rbac-proxy" Jul 02 18:52:27.238257 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.238184 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kube-rbac-proxy" Jul 02 18:52:27.238257 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.238209 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" Jul 02 18:52:27.238257 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.238221 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" Jul 02 18:52:27.238411 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.238320 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kserve-container" Jul 02 18:52:27.238411 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.238335 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="898521dc-c12d-49ba-91bb-d9bf1437cf21" containerName="kube-rbac-proxy" Jul 02 18:52:27.241650 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.241612 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:27.243896 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.243871 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-7acc8-kube-rbac-proxy-sar-config\"" Jul 02 18:52:27.244016 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.243875 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-7acc8-serving-cert\"" Jul 02 18:52:27.245046 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.245024 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm"] Jul 02 18:52:27.339109 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.339070 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7679d6ba-0118-46be-b76d-b37b54205a3a-openshift-service-ca-bundle\") pod \"ensemble-graph-7acc8-6c9bbc6b6f-mqxtm\" (UID: \"7679d6ba-0118-46be-b76d-b37b54205a3a\") " pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:27.339287 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.339136 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7679d6ba-0118-46be-b76d-b37b54205a3a-proxy-tls\") pod \"ensemble-graph-7acc8-6c9bbc6b6f-mqxtm\" (UID: \"7679d6ba-0118-46be-b76d-b37b54205a3a\") " pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:27.440085 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.440046 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7679d6ba-0118-46be-b76d-b37b54205a3a-proxy-tls\") pod \"ensemble-graph-7acc8-6c9bbc6b6f-mqxtm\" (UID: \"7679d6ba-0118-46be-b76d-b37b54205a3a\") " pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:27.440260 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.440136 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7679d6ba-0118-46be-b76d-b37b54205a3a-openshift-service-ca-bundle\") pod \"ensemble-graph-7acc8-6c9bbc6b6f-mqxtm\" (UID: \"7679d6ba-0118-46be-b76d-b37b54205a3a\") " pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:27.440794 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.440775 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7679d6ba-0118-46be-b76d-b37b54205a3a-openshift-service-ca-bundle\") pod \"ensemble-graph-7acc8-6c9bbc6b6f-mqxtm\" (UID: \"7679d6ba-0118-46be-b76d-b37b54205a3a\") " pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:27.442645 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.442606 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7679d6ba-0118-46be-b76d-b37b54205a3a-proxy-tls\") pod \"ensemble-graph-7acc8-6c9bbc6b6f-mqxtm\" (UID: \"7679d6ba-0118-46be-b76d-b37b54205a3a\") " pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:27.553223 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.553133 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:27.688692 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:27.688550 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm"] Jul 02 18:52:27.690805 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:52:27.690777 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7679d6ba_0118_46be_b76d_b37b54205a3a.slice/crio-a28b9f13117a1d1a039405b7cd4022519c659771b9e8a5c8cb43057fd3c98aa7 WatchSource:0}: Error finding container a28b9f13117a1d1a039405b7cd4022519c659771b9e8a5c8cb43057fd3c98aa7: Status 404 returned error can't find the container with id a28b9f13117a1d1a039405b7cd4022519c659771b9e8a5c8cb43057fd3c98aa7 Jul 02 18:52:28.321194 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:28.321152 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:52:28.470893 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:28.470862 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" event={"ID":"7679d6ba-0118-46be-b76d-b37b54205a3a","Type":"ContainerStarted","Data":"5ac09c3d9b6c4b76d59c5189fdf340a1cebde732d03bbc6c5520a6cb63ab8429"} Jul 02 18:52:28.471053 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:28.470913 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" event={"ID":"7679d6ba-0118-46be-b76d-b37b54205a3a","Type":"ContainerStarted","Data":"a28b9f13117a1d1a039405b7cd4022519c659771b9e8a5c8cb43057fd3c98aa7"} Jul 02 18:52:28.471053 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:28.470982 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:28.485073 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:28.485027 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" podStartSLOduration=1.485011396 podStartE2EDuration="1.485011396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:52:28.484071151 +0000 UTC m=+1305.770421008" watchObservedRunningTime="2026-07-02 18:52:28.485011396 +0000 UTC m=+1305.771361239" Jul 02 18:52:33.321237 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:33.321199 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 18:52:34.481106 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:34.481077 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 18:52:35.388268 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:35.388229 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 02 18:52:36.992736 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:36.992705 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:52:37.127209 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.127108 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d06625-9504-4963-a5c6-656c3fc8fb94-proxy-tls\") pod \"d3d06625-9504-4963-a5c6-656c3fc8fb94\" (UID: \"d3d06625-9504-4963-a5c6-656c3fc8fb94\") " Jul 02 18:52:37.127209 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.127166 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3d06625-9504-4963-a5c6-656c3fc8fb94-openshift-service-ca-bundle\") pod \"d3d06625-9504-4963-a5c6-656c3fc8fb94\" (UID: \"d3d06625-9504-4963-a5c6-656c3fc8fb94\") " Jul 02 18:52:37.127519 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.127494 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d06625-9504-4963-a5c6-656c3fc8fb94-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d3d06625-9504-4963-a5c6-656c3fc8fb94" (UID: "d3d06625-9504-4963-a5c6-656c3fc8fb94"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 18:52:37.129169 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.129146 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d06625-9504-4963-a5c6-656c3fc8fb94-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3d06625-9504-4963-a5c6-656c3fc8fb94" (UID: "d3d06625-9504-4963-a5c6-656c3fc8fb94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 18:52:37.228627 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.228587 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d06625-9504-4963-a5c6-656c3fc8fb94-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:52:37.228627 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.228627 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3d06625-9504-4963-a5c6-656c3fc8fb94-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 18:52:37.511866 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.511835 2552 generic.go:356] "Generic (PLEG): container finished" podID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerID="5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195" exitCode=0 Jul 02 18:52:37.512044 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.511890 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" event={"ID":"d3d06625-9504-4963-a5c6-656c3fc8fb94","Type":"ContainerDied","Data":"5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195"} Jul 02 18:52:37.512044 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.511892 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" Jul 02 18:52:37.512044 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.511938 2552 scope.go:117] "RemoveContainer" containerID="5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195" Jul 02 18:52:37.512044 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.511927 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54" event={"ID":"d3d06625-9504-4963-a5c6-656c3fc8fb94","Type":"ContainerDied","Data":"76b647fecbed351b83a4882af4b1749753bad2c97858bfcba25e423efc911a6a"} Jul 02 18:52:37.520119 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.520101 2552 scope.go:117] "RemoveContainer" containerID="5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195" Jul 02 18:52:37.520387 ip-10-0-130-23 kubenswrapper[2552]: E0702 18:52:37.520365 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195\": container with ID starting with 5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195 not found: ID does not exist" containerID="5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195" Jul 02 18:52:37.520485 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.520390 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195"} err="failed to get container status \"5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195\": rpc error: code = NotFound desc = could not find container \"5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195\": container with ID starting with 5fe06307e703faa1e98ae69b1257eb484e38188b4dadef15c8e5b240eead6195 not found: ID does not exist" Jul 02 18:52:37.526417 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.526395 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54"] Jul 02 18:52:37.529742 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:37.529721 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-94e8c-5775877b49-zkm54"] Jul 02 18:52:39.236187 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:39.236149 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" path="/var/lib/kubelet/pods/d3d06625-9504-4963-a5c6-656c3fc8fb94/volumes" Jul 02 18:52:45.388768 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:45.388729 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 02 18:52:55.389718 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:52:55.389689 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 18:53:07.061668 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.061569 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs"] Jul 02 18:53:07.062009 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.061971 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" Jul 02 18:53:07.062009 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.061984 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" Jul 02 18:53:07.062080 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.062043 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d06625-9504-4963-a5c6-656c3fc8fb94" containerName="sequence-graph-94e8c" Jul 02 18:53:07.066431 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.066412 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 18:53:07.068512 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.068491 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ae135-serving-cert\"" Jul 02 18:53:07.068599 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.068509 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ae135-kube-rbac-proxy-sar-config\"" Jul 02 18:53:07.070852 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.070831 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs"] Jul 02 18:53:07.178155 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.178120 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-proxy-tls\") pod \"sequence-graph-ae135-78dffc7b7d-86kjs\" (UID: \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\") " pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 18:53:07.178327 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.178181 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-openshift-service-ca-bundle\") pod \"sequence-graph-ae135-78dffc7b7d-86kjs\" (UID: \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\") " pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 18:53:07.278852 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.278817 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-openshift-service-ca-bundle\") pod \"sequence-graph-ae135-78dffc7b7d-86kjs\" (UID: \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\") " pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 18:53:07.279017 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.278891 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-proxy-tls\") pod \"sequence-graph-ae135-78dffc7b7d-86kjs\" (UID: \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\") " pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 18:53:07.279500 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.279479 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-openshift-service-ca-bundle\") pod \"sequence-graph-ae135-78dffc7b7d-86kjs\" (UID: \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\") " pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 18:53:07.281362 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.281337 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-proxy-tls\") pod \"sequence-graph-ae135-78dffc7b7d-86kjs\" (UID: \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\") " pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 18:53:07.377272 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.377190 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 18:53:07.505178 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.505032 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs"] Jul 02 18:53:07.507421 ip-10-0-130-23 kubenswrapper[2552]: W0702 18:53:07.507391 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9f4197_c9d1_48b1_b605_52b58b3e8e59.slice/crio-b42525de94330983194876c72801525e5e5f06e188cd45cb4194c5ac3aee729d WatchSource:0}: Error finding container b42525de94330983194876c72801525e5e5f06e188cd45cb4194c5ac3aee729d: Status 404 returned error can't find the container with id b42525de94330983194876c72801525e5e5f06e188cd45cb4194c5ac3aee729d Jul 02 18:53:07.646780 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.646694 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" event={"ID":"ef9f4197-c9d1-48b1-b605-52b58b3e8e59","Type":"ContainerStarted","Data":"7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a"} Jul 02 18:53:07.646780 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.646747 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" event={"ID":"ef9f4197-c9d1-48b1-b605-52b58b3e8e59","Type":"ContainerStarted","Data":"b42525de94330983194876c72801525e5e5f06e188cd45cb4194c5ac3aee729d"} Jul 02 18:53:07.646975 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.646812 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 18:53:07.663526 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:07.663472 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" podStartSLOduration=0.663456745 podStartE2EDuration="663.456745ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:53:07.661710902 +0000 UTC m=+1344.948060745" watchObservedRunningTime="2026-07-02 18:53:07.663456745 +0000 UTC m=+1344.949806588" Jul 02 18:53:13.657001 ip-10-0-130-23 kubenswrapper[2552]: I0702 18:53:13.656975 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 19:00:41.755382 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.755349 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm"] Jul 02 19:00:41.755867 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.755561 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" containerID="cri-o://5ac09c3d9b6c4b76d59c5189fdf340a1cebde732d03bbc6c5520a6cb63ab8429" gracePeriod=30 Jul 02 19:00:41.967996 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.967952 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv"] Jul 02 19:00:41.972171 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.972150 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf"] Jul 02 19:00:41.972293 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.972259 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:41.972334 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.972315 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" containerID="cri-o://a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e" gracePeriod=30 Jul 02 19:00:41.972551 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.972439 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kube-rbac-proxy" containerID="cri-o://0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861" gracePeriod=30 Jul 02 19:00:41.974323 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.974300 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c532a-kube-rbac-proxy-sar-config\"" Jul 02 19:00:41.974449 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.974300 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c532a-predictor-serving-cert\"" Jul 02 19:00:41.981391 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:41.981370 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv"] Jul 02 19:00:42.057093 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.057016 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.057093 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.057062 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcad90-e414-454f-bf37-f2c6d826c0f1-proxy-tls\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.057093 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.057086 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4qx\" (UniqueName: \"kubernetes.io/projected/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-api-access-qn4qx\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.158491 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.158454 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.158491 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.158496 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcad90-e414-454f-bf37-f2c6d826c0f1-proxy-tls\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.158789 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.158516 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4qx\" (UniqueName: \"kubernetes.io/projected/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-api-access-qn4qx\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.158789 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:00:42.158631 2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c532a-predictor-serving-cert: secret "success-200-isvc-c532a-predictor-serving-cert" not found Jul 02 19:00:42.158789 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:00:42.158737 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dcad90-e414-454f-bf37-f2c6d826c0f1-proxy-tls podName:d6dcad90-e414-454f-bf37-f2c6d826c0f1 nodeName:}" failed. No retries permitted until 2026-07-02 19:00:42.658714922 +0000 UTC m=+1799.945064757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d6dcad90-e414-454f-bf37-f2c6d826c0f1-proxy-tls") pod "success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" (UID: "d6dcad90-e414-454f-bf37-f2c6d826c0f1") : secret "success-200-isvc-c532a-predictor-serving-cert" not found Jul 02 19:00:42.159167 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.159150 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.168507 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.168475 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4qx\" (UniqueName: \"kubernetes.io/projected/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-api-access-qn4qx\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.663984 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.663945 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcad90-e414-454f-bf37-f2c6d826c0f1-proxy-tls\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.666470 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.666428 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcad90-e414-454f-bf37-f2c6d826c0f1-proxy-tls\") pod \"success-200-isvc-c532a-predictor-6988ddc98f-c4fgv\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:42.720192 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.720162 2552 generic.go:356] "Generic (PLEG): container finished" podID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerID="0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861" exitCode=2 Jul 02 19:00:42.720334 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.720234 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" event={"ID":"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb","Type":"ContainerDied","Data":"0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861"} Jul 02 19:00:42.893745 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:42.893716 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:43.023082 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:43.023044 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv"] Jul 02 19:00:43.025590 ip-10-0-130-23 kubenswrapper[2552]: W0702 19:00:43.025546 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dcad90_e414_454f_bf37_f2c6d826c0f1.slice/crio-fc08b888a04f761b03634cbdb1b402b536e90be1578e8df0017703dd699cfbb1 WatchSource:0}: Error finding container fc08b888a04f761b03634cbdb1b402b536e90be1578e8df0017703dd699cfbb1: Status 404 returned error can't find the container with id fc08b888a04f761b03634cbdb1b402b536e90be1578e8df0017703dd699cfbb1 Jul 02 19:00:43.726905 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:43.726867 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" event={"ID":"d6dcad90-e414-454f-bf37-f2c6d826c0f1","Type":"ContainerStarted","Data":"e081cec2f245702908087fa0207e38c635bf8568bfc8ac6918ee45da6c201219"} Jul 02 19:00:43.727107 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:43.726933 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" event={"ID":"d6dcad90-e414-454f-bf37-f2c6d826c0f1","Type":"ContainerStarted","Data":"59bba5e3f4c7038d145b5188d69ba727d881fb329c17369a23192d1a47ecccc1"} Jul 02 19:00:43.727107 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:43.726961 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" event={"ID":"d6dcad90-e414-454f-bf37-f2c6d826c0f1","Type":"ContainerStarted","Data":"fc08b888a04f761b03634cbdb1b402b536e90be1578e8df0017703dd699cfbb1"} Jul 02 19:00:43.727107 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:43.726992 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:43.727280 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:43.727154 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:43.728572 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:43.728550 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 02 19:00:43.742996 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:43.742944 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podStartSLOduration=2.742931664 podStartE2EDuration="2.742931664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:00:43.740991892 +0000 UTC m=+1801.027341735" watchObservedRunningTime="2026-07-02 19:00:43.742931664 +0000 UTC m=+1801.029281507" Jul 02 19:00:44.479609 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:44.479569 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:00:44.733408 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:44.733305 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 02 19:00:45.225327 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.225305 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 19:00:45.285503 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.285467 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-proxy-tls\") pod \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " Jul 02 19:00:45.285503 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.285504 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-rbac-proxy-sar-config\") pod \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " Jul 02 19:00:45.285760 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.285671 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79sbt\" (UniqueName: \"kubernetes.io/projected/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-api-access-79sbt\") pod \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\" (UID: \"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb\") " Jul 02 19:00:45.285933 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.285908 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" (UID: "fffc5bbc-f8c3-4ba0-839c-d20e01d598bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:00:45.286014 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.285989 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:00:45.287568 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.287550 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" (UID: "fffc5bbc-f8c3-4ba0-839c-d20e01d598bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:00:45.287770 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.287748 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-api-access-79sbt" (OuterVolumeSpecName: "kube-api-access-79sbt") pod "fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" (UID: "fffc5bbc-f8c3-4ba0-839c-d20e01d598bb"). InnerVolumeSpecName "kube-api-access-79sbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:00:45.386873 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.386788 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:00:45.386873 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.386817 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79sbt\" (UniqueName: \"kubernetes.io/projected/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb-kube-api-access-79sbt\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:00:45.739495 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.739459 2552 generic.go:356] "Generic (PLEG): container finished" podID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerID="a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e" exitCode=0 Jul 02 19:00:45.740007 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.739505 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" Jul 02 19:00:45.740007 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.739497 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" event={"ID":"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb","Type":"ContainerDied","Data":"a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e"} Jul 02 19:00:45.740007 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.739670 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" event={"ID":"fffc5bbc-f8c3-4ba0-839c-d20e01d598bb","Type":"ContainerDied","Data":"cb46905146c0f1fe626e6dc72a7e5701f3c98765055800fe78bce8de9fe6beda"} Jul 02 19:00:45.740007 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.739720 2552 scope.go:117] "RemoveContainer" containerID="0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861" Jul 02 19:00:45.748624 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.748609 2552 scope.go:117] "RemoveContainer" containerID="a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e" Jul 02 19:00:45.756135 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.756120 2552 scope.go:117] "RemoveContainer" containerID="0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861" Jul 02 19:00:45.756372 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:00:45.756353 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861\": container with ID starting with 0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861 not found: ID does not exist" containerID="0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861" Jul 02 19:00:45.756415 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.756380 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861"} err="failed to get container status \"0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861\": rpc error: code = NotFound desc = could not find container \"0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861\": container with ID starting with 0158c7edb197eb8af2c7035d27c49a5d14ff6fffa88d218c07e09ec3606d9861 not found: ID does not exist" Jul 02 19:00:45.756415 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.756396 2552 scope.go:117] "RemoveContainer" containerID="a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e" Jul 02 19:00:45.756573 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:00:45.756557 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e\": container with ID starting with a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e not found: ID does not exist" containerID="a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e" Jul 02 19:00:45.756613 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.756579 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e"} err="failed to get container status \"a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e\": rpc error: code = NotFound desc = could not find container \"a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e\": container with ID starting with a01d92b06dc179c52dd4408ba671dd6e8a85dcd04cace3868cdb911c29b1822e not found: ID does not exist" Jul 02 19:00:45.760044 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.760022 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf"] Jul 02 19:00:45.763178 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:45.763157 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf"] Jul 02 19:00:46.178965 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:46.178919 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": context deadline exceeded" Jul 02 19:00:46.183500 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:46.183461 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7acc8-predictor-5896b7c7c-74fgf" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: i/o timeout" Jul 02 19:00:47.236717 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:47.236683 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" path="/var/lib/kubelet/pods/fffc5bbc-f8c3-4ba0-839c-d20e01d598bb/volumes" Jul 02 19:00:49.479560 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:49.479517 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:00:49.738030 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:49.737934 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:00:49.738488 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:49.738459 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 02 19:00:54.480470 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:54.480434 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:00:54.480854 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:54.480532 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 19:00:59.479798 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:59.479762 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:00:59.739155 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:00:59.739070 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 02 19:01:04.480315 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:04.480275 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:01:09.479786 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:09.479746 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:01:09.738568 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:09.738481 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 02 19:01:11.862384 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:11.862345 2552 generic.go:356] "Generic (PLEG): container finished" podID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerID="5ac09c3d9b6c4b76d59c5189fdf340a1cebde732d03bbc6c5520a6cb63ab8429" exitCode=0 Jul 02 19:01:11.862793 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:11.862415 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" event={"ID":"7679d6ba-0118-46be-b76d-b37b54205a3a","Type":"ContainerDied","Data":"5ac09c3d9b6c4b76d59c5189fdf340a1cebde732d03bbc6c5520a6cb63ab8429"} Jul 02 19:01:11.906166 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:11.906145 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 19:01:11.924667 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:11.924619 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7679d6ba-0118-46be-b76d-b37b54205a3a-openshift-service-ca-bundle\") pod \"7679d6ba-0118-46be-b76d-b37b54205a3a\" (UID: \"7679d6ba-0118-46be-b76d-b37b54205a3a\") " Jul 02 19:01:11.924821 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:11.924722 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7679d6ba-0118-46be-b76d-b37b54205a3a-proxy-tls\") pod \"7679d6ba-0118-46be-b76d-b37b54205a3a\" (UID: \"7679d6ba-0118-46be-b76d-b37b54205a3a\") " Jul 02 19:01:11.925001 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:11.924978 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7679d6ba-0118-46be-b76d-b37b54205a3a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7679d6ba-0118-46be-b76d-b37b54205a3a" (UID: "7679d6ba-0118-46be-b76d-b37b54205a3a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:01:11.926962 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:11.926936 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7679d6ba-0118-46be-b76d-b37b54205a3a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7679d6ba-0118-46be-b76d-b37b54205a3a" (UID: "7679d6ba-0118-46be-b76d-b37b54205a3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:01:12.026052 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:12.025972 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7679d6ba-0118-46be-b76d-b37b54205a3a-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:12.026052 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:12.025999 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7679d6ba-0118-46be-b76d-b37b54205a3a-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:12.869149 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:12.869110 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" event={"ID":"7679d6ba-0118-46be-b76d-b37b54205a3a","Type":"ContainerDied","Data":"a28b9f13117a1d1a039405b7cd4022519c659771b9e8a5c8cb43057fd3c98aa7"} Jul 02 19:01:12.869599 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:12.869172 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm" Jul 02 19:01:12.869599 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:12.869188 2552 scope.go:117] "RemoveContainer" containerID="5ac09c3d9b6c4b76d59c5189fdf340a1cebde732d03bbc6c5520a6cb63ab8429" Jul 02 19:01:12.889680 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:12.889655 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm"] Jul 02 19:01:12.893418 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:12.893398 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7acc8-6c9bbc6b6f-mqxtm"] Jul 02 19:01:13.237589 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:13.237556 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" path="/var/lib/kubelet/pods/7679d6ba-0118-46be-b76d-b37b54205a3a/volumes" Jul 02 19:01:19.738756 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:19.738715 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 02 19:01:21.829236 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.829199 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs"] Jul 02 19:01:21.829702 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.829439 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" containerID="cri-o://7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a" gracePeriod=30 Jul 02 19:01:21.964513 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.964478 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz"] Jul 02 19:01:21.964780 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.964756 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" containerID="cri-o://9551617cda14d58fb9f115ebc280c247d2f8d9b2dc148bc4da778ad37d2cd706" gracePeriod=30 Jul 02 19:01:21.964873 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.964848 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kube-rbac-proxy" containerID="cri-o://289dcb6420212f769c4d46f61353e233973c402945f57c06f914161c91dd79b4" gracePeriod=30 Jul 02 19:01:21.986804 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.986763 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7"] Jul 02 19:01:21.987399 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.987378 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" Jul 02 19:01:21.987460 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.987403 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" Jul 02 19:01:21.987460 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.987418 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" Jul 02 19:01:21.987460 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.987425 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" Jul 02 19:01:21.987460 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.987436 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kube-rbac-proxy" Jul 02 19:01:21.987460 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.987442 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kube-rbac-proxy" Jul 02 19:01:21.987619 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.987516 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kserve-container" Jul 02 19:01:21.987619 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.987526 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffc5bbc-f8c3-4ba0-839c-d20e01d598bb" containerName="kube-rbac-proxy" Jul 02 19:01:21.987619 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.987531 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7679d6ba-0118-46be-b76d-b37b54205a3a" containerName="ensemble-graph-7acc8" Jul 02 19:01:21.990679 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.990657 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:21.992522 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.992502 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-208dc-predictor-serving-cert\"" Jul 02 19:01:21.992608 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.992512 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-208dc-kube-rbac-proxy-sar-config\"" Jul 02 19:01:21.997944 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:21.997881 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7"] Jul 02 19:01:22.108686 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.108656 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.108832 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.108748 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b39e9-fa02-4592-8f70-01d5bd2bacbf-proxy-tls\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.108832 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.108784 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqjk\" (UniqueName: \"kubernetes.io/projected/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-api-access-jhqjk\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.209926 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.209896 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.210109 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.209954 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b39e9-fa02-4592-8f70-01d5bd2bacbf-proxy-tls\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.210109 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.209983 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqjk\" (UniqueName: \"kubernetes.io/projected/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-api-access-jhqjk\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.210246 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:01:22.210121 2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-208dc-predictor-serving-cert: secret "success-200-isvc-208dc-predictor-serving-cert" not found Jul 02 19:01:22.210246 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:01:22.210191 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/704b39e9-fa02-4592-8f70-01d5bd2bacbf-proxy-tls podName:704b39e9-fa02-4592-8f70-01d5bd2bacbf nodeName:}" failed. No retries permitted until 2026-07-02 19:01:22.710172467 +0000 UTC m=+1839.996522296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/704b39e9-fa02-4592-8f70-01d5bd2bacbf-proxy-tls") pod "success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" (UID: "704b39e9-fa02-4592-8f70-01d5bd2bacbf") : secret "success-200-isvc-208dc-predictor-serving-cert" not found Jul 02 19:01:22.210661 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.210611 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.217877 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.217856 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqjk\" (UniqueName: \"kubernetes.io/projected/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-api-access-jhqjk\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.714681 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.714650 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b39e9-fa02-4592-8f70-01d5bd2bacbf-proxy-tls\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.716973 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.716954 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b39e9-fa02-4592-8f70-01d5bd2bacbf-proxy-tls\") pod \"success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.901290 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.901259 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:22.919504 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.919477 2552 generic.go:356] "Generic (PLEG): container finished" podID="533fd46b-98d4-4392-975d-79511afd23fb" containerID="289dcb6420212f769c4d46f61353e233973c402945f57c06f914161c91dd79b4" exitCode=2 Jul 02 19:01:22.919607 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:22.919523 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" event={"ID":"533fd46b-98d4-4392-975d-79511afd23fb","Type":"ContainerDied","Data":"289dcb6420212f769c4d46f61353e233973c402945f57c06f914161c91dd79b4"} Jul 02 19:01:23.031486 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:23.031462 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7"] Jul 02 19:01:23.033838 ip-10-0-130-23 kubenswrapper[2552]: W0702 19:01:23.033811 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704b39e9_fa02_4592_8f70_01d5bd2bacbf.slice/crio-e88a401ffcfbf32faeaecfe30e807d79027a4909af00274543df1dc85ef6b2e5 WatchSource:0}: Error finding container e88a401ffcfbf32faeaecfe30e807d79027a4909af00274543df1dc85ef6b2e5: Status 404 returned error can't find the container with id e88a401ffcfbf32faeaecfe30e807d79027a4909af00274543df1dc85ef6b2e5 Jul 02 19:01:23.656061 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:23.656025 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:01:23.929762 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:23.929722 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" event={"ID":"704b39e9-fa02-4592-8f70-01d5bd2bacbf","Type":"ContainerStarted","Data":"10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1"} Jul 02 19:01:23.930220 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:23.929841 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" event={"ID":"704b39e9-fa02-4592-8f70-01d5bd2bacbf","Type":"ContainerStarted","Data":"7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f"} Jul 02 19:01:23.930220 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:23.929883 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" event={"ID":"704b39e9-fa02-4592-8f70-01d5bd2bacbf","Type":"ContainerStarted","Data":"e88a401ffcfbf32faeaecfe30e807d79027a4909af00274543df1dc85ef6b2e5"} Jul 02 19:01:23.931166 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:23.930731 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:23.931166 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:23.930819 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:23.932346 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:23.932316 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 02 19:01:23.947566 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:23.947522 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podStartSLOduration=2.947509351 podStartE2EDuration="2.947509351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:01:23.94594393 +0000 UTC m=+1841.232293775" watchObservedRunningTime="2026-07-02 19:01:23.947509351 +0000 UTC m=+1841.233859194" Jul 02 19:01:24.936801 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:24.936772 2552 generic.go:356] "Generic (PLEG): container finished" podID="533fd46b-98d4-4392-975d-79511afd23fb" containerID="9551617cda14d58fb9f115ebc280c247d2f8d9b2dc148bc4da778ad37d2cd706" exitCode=0 Jul 02 19:01:24.937110 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:24.936849 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" event={"ID":"533fd46b-98d4-4392-975d-79511afd23fb","Type":"ContainerDied","Data":"9551617cda14d58fb9f115ebc280c247d2f8d9b2dc148bc4da778ad37d2cd706"} Jul 02 19:01:24.937110 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:24.937078 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 02 19:01:25.014721 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.014700 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 19:01:25.137120 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.137051 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pccf\" (UniqueName: \"kubernetes.io/projected/533fd46b-98d4-4392-975d-79511afd23fb-kube-api-access-8pccf\") pod \"533fd46b-98d4-4392-975d-79511afd23fb\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " Jul 02 19:01:25.137120 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.137097 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533fd46b-98d4-4392-975d-79511afd23fb-proxy-tls\") pod \"533fd46b-98d4-4392-975d-79511afd23fb\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " Jul 02 19:01:25.137285 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.137166 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/533fd46b-98d4-4392-975d-79511afd23fb-kube-rbac-proxy-sar-config\") pod \"533fd46b-98d4-4392-975d-79511afd23fb\" (UID: \"533fd46b-98d4-4392-975d-79511afd23fb\") " Jul 02 19:01:25.137563 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.137535 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533fd46b-98d4-4392-975d-79511afd23fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "533fd46b-98d4-4392-975d-79511afd23fb" (UID: "533fd46b-98d4-4392-975d-79511afd23fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:01:25.139215 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.139182 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533fd46b-98d4-4392-975d-79511afd23fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "533fd46b-98d4-4392-975d-79511afd23fb" (UID: "533fd46b-98d4-4392-975d-79511afd23fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:01:25.139215 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.139202 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533fd46b-98d4-4392-975d-79511afd23fb-kube-api-access-8pccf" (OuterVolumeSpecName: "kube-api-access-8pccf") pod "533fd46b-98d4-4392-975d-79511afd23fb" (UID: "533fd46b-98d4-4392-975d-79511afd23fb"). InnerVolumeSpecName "kube-api-access-8pccf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:01:25.237815 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.237787 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/533fd46b-98d4-4392-975d-79511afd23fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:25.237815 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.237811 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pccf\" (UniqueName: \"kubernetes.io/projected/533fd46b-98d4-4392-975d-79511afd23fb-kube-api-access-8pccf\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:25.237980 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.237821 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533fd46b-98d4-4392-975d-79511afd23fb-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:25.942949 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.942923 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" Jul 02 19:01:25.943360 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.942962 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz" event={"ID":"533fd46b-98d4-4392-975d-79511afd23fb","Type":"ContainerDied","Data":"eba9b1d7bf718a5e2861fd78f5d3b2fecf7e07a62d7b8e86cb359be506cea10a"} Jul 02 19:01:25.943360 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.943029 2552 scope.go:117] "RemoveContainer" containerID="289dcb6420212f769c4d46f61353e233973c402945f57c06f914161c91dd79b4" Jul 02 19:01:25.943528 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.943485 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 02 19:01:25.951414 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.951394 2552 scope.go:117] "RemoveContainer" containerID="9551617cda14d58fb9f115ebc280c247d2f8d9b2dc148bc4da778ad37d2cd706" Jul 02 19:01:25.961257 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.961230 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz"] Jul 02 19:01:25.963141 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:25.963117 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae135-predictor-686c9c7795-6p9jz"] Jul 02 19:01:27.236261 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:27.236220 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533fd46b-98d4-4392-975d-79511afd23fb" path="/var/lib/kubelet/pods/533fd46b-98d4-4392-975d-79511afd23fb/volumes" Jul 02 19:01:28.655161 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:28.655126 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:01:29.739778 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:29.739745 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:01:30.948509 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:30.948478 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:01:30.948960 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:30.948933 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 02 19:01:33.655555 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:33.655520 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:01:33.655970 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:33.655620 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 19:01:38.656839 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:38.656798 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:01:40.949676 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:40.949620 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 02 19:01:41.977058 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.977027 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb"] Jul 02 19:01:41.977446 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.977416 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kube-rbac-proxy" Jul 02 19:01:41.977446 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.977428 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kube-rbac-proxy" Jul 02 19:01:41.977446 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.977444 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" Jul 02 19:01:41.977553 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.977449 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" Jul 02 19:01:41.977553 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.977509 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kserve-container" Jul 02 19:01:41.977553 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.977517 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="533fd46b-98d4-4392-975d-79511afd23fb" containerName="kube-rbac-proxy" Jul 02 19:01:41.980616 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.980601 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:41.982541 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.982483 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-c532a-kube-rbac-proxy-sar-config\"" Jul 02 19:01:41.982703 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.982503 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-c532a-serving-cert\"" Jul 02 19:01:41.987497 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:41.987472 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb"] Jul 02 19:01:42.084184 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:42.084142 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61d3472c-ea65-43b2-beaf-387cdbb7ea44-openshift-service-ca-bundle\") pod \"splitter-graph-c532a-76d7947565-k9nvb\" (UID: \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\") " pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:42.084361 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:42.084226 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61d3472c-ea65-43b2-beaf-387cdbb7ea44-proxy-tls\") pod \"splitter-graph-c532a-76d7947565-k9nvb\" (UID: \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\") " pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:42.184859 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:42.184817 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61d3472c-ea65-43b2-beaf-387cdbb7ea44-openshift-service-ca-bundle\") pod \"splitter-graph-c532a-76d7947565-k9nvb\" (UID: \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\") " pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:42.185038 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:42.184879 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61d3472c-ea65-43b2-beaf-387cdbb7ea44-proxy-tls\") pod \"splitter-graph-c532a-76d7947565-k9nvb\" (UID: \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\") " pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:42.185503 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:42.185473 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61d3472c-ea65-43b2-beaf-387cdbb7ea44-openshift-service-ca-bundle\") pod \"splitter-graph-c532a-76d7947565-k9nvb\" (UID: \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\") " pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:42.187200 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:42.187183 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61d3472c-ea65-43b2-beaf-387cdbb7ea44-proxy-tls\") pod \"splitter-graph-c532a-76d7947565-k9nvb\" (UID: \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\") " pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:42.292270 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:42.292205 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:42.419507 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:42.419483 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb"] Jul 02 19:01:42.421408 ip-10-0-130-23 kubenswrapper[2552]: W0702 19:01:42.421378 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d3472c_ea65_43b2_beaf_387cdbb7ea44.slice/crio-6e1c241312a7040940c25df7aa64a291b30b3380a8b7161cf297be3671d16436 WatchSource:0}: Error finding container 6e1c241312a7040940c25df7aa64a291b30b3380a8b7161cf297be3671d16436: Status 404 returned error can't find the container with id 6e1c241312a7040940c25df7aa64a291b30b3380a8b7161cf297be3671d16436 Jul 02 19:01:43.027392 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:43.027359 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" event={"ID":"61d3472c-ea65-43b2-beaf-387cdbb7ea44","Type":"ContainerStarted","Data":"df7e2524c87b87009792a8235777bd65fa016b651ad7a4f8d6f0eb87713b2762"} Jul 02 19:01:43.027878 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:43.027418 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:43.027878 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:43.027438 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" event={"ID":"61d3472c-ea65-43b2-beaf-387cdbb7ea44","Type":"ContainerStarted","Data":"6e1c241312a7040940c25df7aa64a291b30b3380a8b7161cf297be3671d16436"} Jul 02 19:01:43.044732 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:43.044670 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" podStartSLOduration=2.044651407 podStartE2EDuration="2.044651407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:01:43.040821491 +0000 UTC m=+1860.327171334" watchObservedRunningTime="2026-07-02 19:01:43.044651407 +0000 UTC m=+1860.331001247" Jul 02 19:01:43.655729 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:43.655624 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:01:48.655499 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:48.655460 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:01:49.038302 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:49.038273 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:01:50.949390 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:50.949351 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 02 19:01:51.976664 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:51.976626 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 19:01:52.059447 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.059372 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb"] Jul 02 19:01:52.059587 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.059567 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" containerID="cri-o://df7e2524c87b87009792a8235777bd65fa016b651ad7a4f8d6f0eb87713b2762" gracePeriod=30 Jul 02 19:01:52.071574 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.071540 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-openshift-service-ca-bundle\") pod \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\" (UID: \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\") " Jul 02 19:01:52.071739 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.071579 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-proxy-tls\") pod \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\" (UID: \"ef9f4197-c9d1-48b1-b605-52b58b3e8e59\") " Jul 02 19:01:52.071987 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.071960 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ef9f4197-c9d1-48b1-b605-52b58b3e8e59" (UID: "ef9f4197-c9d1-48b1-b605-52b58b3e8e59"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:01:52.073892 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.073856 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef9f4197-c9d1-48b1-b605-52b58b3e8e59" (UID: "ef9f4197-c9d1-48b1-b605-52b58b3e8e59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:01:52.077977 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.077946 2552 generic.go:356] "Generic (PLEG): container finished" podID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerID="7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a" exitCode=0 Jul 02 19:01:52.078122 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.077996 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" Jul 02 19:01:52.078122 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.078021 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" event={"ID":"ef9f4197-c9d1-48b1-b605-52b58b3e8e59","Type":"ContainerDied","Data":"7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a"} Jul 02 19:01:52.078122 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.078082 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs" event={"ID":"ef9f4197-c9d1-48b1-b605-52b58b3e8e59","Type":"ContainerDied","Data":"b42525de94330983194876c72801525e5e5f06e188cd45cb4194c5ac3aee729d"} Jul 02 19:01:52.078122 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.078113 2552 scope.go:117] "RemoveContainer" containerID="7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a" Jul 02 19:01:52.088572 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.088553 2552 scope.go:117] "RemoveContainer" containerID="7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a" Jul 02 19:01:52.088834 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:01:52.088817 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a\": container with ID starting with 7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a not found: ID does not exist" containerID="7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a" Jul 02 19:01:52.088889 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.088842 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a"} err="failed to get container status \"7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a\": rpc error: code = NotFound desc = could not find container \"7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a\": container with ID starting with 7796ed5725b8c54463bce7acd61720d9cb34eb81b24950e60cc1256a4fd2d90a not found: ID does not exist" Jul 02 19:01:52.101143 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.101121 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs"] Jul 02 19:01:52.106503 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.106483 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ae135-78dffc7b7d-86kjs"] Jul 02 19:01:52.172765 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.172684 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:52.172765 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.172714 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9f4197-c9d1-48b1-b605-52b58b3e8e59-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:52.186162 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.186133 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv"] Jul 02 19:01:52.186391 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.186363 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" containerID="cri-o://59bba5e3f4c7038d145b5188d69ba727d881fb329c17369a23192d1a47ecccc1" gracePeriod=30 Jul 02 19:01:52.186515 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.186408 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kube-rbac-proxy" containerID="cri-o://e081cec2f245702908087fa0207e38c635bf8568bfc8ac6918ee45da6c201219" gracePeriod=30 Jul 02 19:01:52.203401 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.203378 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt"] Jul 02 19:01:52.203789 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.203774 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" Jul 02 19:01:52.203789 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.203790 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" Jul 02 19:01:52.203918 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.203866 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" containerName="sequence-graph-ae135" Jul 02 19:01:52.208056 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.208037 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.210392 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.210014 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c41f9-kube-rbac-proxy-sar-config\"" Jul 02 19:01:52.210392 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.210072 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c41f9-predictor-serving-cert\"" Jul 02 19:01:52.215459 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.215440 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt"] Jul 02 19:01:52.374699 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.374607 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vtd\" (UniqueName: \"kubernetes.io/projected/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-api-access-d2vtd\") pod \"success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.374699 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.374693 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f316cb-9604-4d0e-b80e-a5c151e307ae-proxy-tls\") pod \"success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.374869 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.374788 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.475835 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.475805 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.475991 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.475857 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vtd\" (UniqueName: \"kubernetes.io/projected/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-api-access-d2vtd\") pod \"success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.475991 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.475890 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f316cb-9604-4d0e-b80e-a5c151e307ae-proxy-tls\") pod \"success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.476429 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.476408 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.478263 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.478239 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f316cb-9604-4d0e-b80e-a5c151e307ae-proxy-tls\") pod \"success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.482765 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.482748 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vtd\" (UniqueName: \"kubernetes.io/projected/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-api-access-d2vtd\") pod \"success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.519784 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.519761 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:52.652735 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:52.652693 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt"] Jul 02 19:01:52.655043 ip-10-0-130-23 kubenswrapper[2552]: W0702 19:01:52.655012 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f316cb_9604_4d0e_b80e_a5c151e307ae.slice/crio-7c9a4b38234dadf6d301e68c3ae08b9db8246be002d78fefed4d688f86fc498f WatchSource:0}: Error finding container 7c9a4b38234dadf6d301e68c3ae08b9db8246be002d78fefed4d688f86fc498f: Status 404 returned error can't find the container with id 7c9a4b38234dadf6d301e68c3ae08b9db8246be002d78fefed4d688f86fc498f Jul 02 19:01:53.085027 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:53.084995 2552 generic.go:356] "Generic (PLEG): container finished" podID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerID="e081cec2f245702908087fa0207e38c635bf8568bfc8ac6918ee45da6c201219" exitCode=2 Jul 02 19:01:53.085432 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:53.085060 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" event={"ID":"d6dcad90-e414-454f-bf37-f2c6d826c0f1","Type":"ContainerDied","Data":"e081cec2f245702908087fa0207e38c635bf8568bfc8ac6918ee45da6c201219"} Jul 02 19:01:53.086773 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:53.086751 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" event={"ID":"29f316cb-9604-4d0e-b80e-a5c151e307ae","Type":"ContainerStarted","Data":"ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85"} Jul 02 19:01:53.086889 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:53.086802 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" event={"ID":"29f316cb-9604-4d0e-b80e-a5c151e307ae","Type":"ContainerStarted","Data":"7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d"} Jul 02 19:01:53.086889 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:53.086824 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" event={"ID":"29f316cb-9604-4d0e-b80e-a5c151e307ae","Type":"ContainerStarted","Data":"7c9a4b38234dadf6d301e68c3ae08b9db8246be002d78fefed4d688f86fc498f"} Jul 02 19:01:53.086963 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:53.086887 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:53.104464 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:53.104420 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podStartSLOduration=1.104406603 podStartE2EDuration="1.104406603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:01:53.10255875 +0000 UTC m=+1870.388908593" watchObservedRunningTime="2026-07-02 19:01:53.104406603 +0000 UTC m=+1870.390756446" Jul 02 19:01:53.236606 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:53.236575 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9f4197-c9d1-48b1-b605-52b58b3e8e59" path="/var/lib/kubelet/pods/ef9f4197-c9d1-48b1-b605-52b58b3e8e59/volumes" Jul 02 19:01:54.036534 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:54.036501 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:01:54.091552 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:54.091517 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:01:54.092997 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:54.092969 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 02 19:01:54.734521 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:54.734480 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused" Jul 02 19:01:55.098778 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.098749 2552 generic.go:356] "Generic (PLEG): container finished" podID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerID="59bba5e3f4c7038d145b5188d69ba727d881fb329c17369a23192d1a47ecccc1" exitCode=0 Jul 02 19:01:55.099112 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.098815 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" event={"ID":"d6dcad90-e414-454f-bf37-f2c6d826c0f1","Type":"ContainerDied","Data":"59bba5e3f4c7038d145b5188d69ba727d881fb329c17369a23192d1a47ecccc1"} Jul 02 19:01:55.099206 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.099182 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 02 19:01:55.138682 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.138661 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:01:55.201543 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.201514 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcad90-e414-454f-bf37-f2c6d826c0f1-proxy-tls\") pod \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " Jul 02 19:01:55.201742 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.201548 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-rbac-proxy-sar-config\") pod \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " Jul 02 19:01:55.201742 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.201568 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn4qx\" (UniqueName: \"kubernetes.io/projected/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-api-access-qn4qx\") pod \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\" (UID: \"d6dcad90-e414-454f-bf37-f2c6d826c0f1\") " Jul 02 19:01:55.201944 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.201920 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6dcad90-e414-454f-bf37-f2c6d826c0f1" (UID: "d6dcad90-e414-454f-bf37-f2c6d826c0f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:01:55.203591 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.203569 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6dcad90-e414-454f-bf37-f2c6d826c0f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6dcad90-e414-454f-bf37-f2c6d826c0f1" (UID: "d6dcad90-e414-454f-bf37-f2c6d826c0f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:01:55.203669 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.203606 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-api-access-qn4qx" (OuterVolumeSpecName: "kube-api-access-qn4qx") pod "d6dcad90-e414-454f-bf37-f2c6d826c0f1" (UID: "d6dcad90-e414-454f-bf37-f2c6d826c0f1"). InnerVolumeSpecName "kube-api-access-qn4qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:01:55.302168 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.302095 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcad90-e414-454f-bf37-f2c6d826c0f1-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:55.302168 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.302123 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:55.302168 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:55.302136 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4qx\" (UniqueName: \"kubernetes.io/projected/d6dcad90-e414-454f-bf37-f2c6d826c0f1-kube-api-access-qn4qx\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:01:56.105202 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:56.105177 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" Jul 02 19:01:56.105602 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:56.105176 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv" event={"ID":"d6dcad90-e414-454f-bf37-f2c6d826c0f1","Type":"ContainerDied","Data":"fc08b888a04f761b03634cbdb1b402b536e90be1578e8df0017703dd699cfbb1"} Jul 02 19:01:56.105602 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:56.105313 2552 scope.go:117] "RemoveContainer" containerID="e081cec2f245702908087fa0207e38c635bf8568bfc8ac6918ee45da6c201219" Jul 02 19:01:56.113852 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:56.113833 2552 scope.go:117] "RemoveContainer" containerID="59bba5e3f4c7038d145b5188d69ba727d881fb329c17369a23192d1a47ecccc1" Jul 02 19:01:56.120946 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:56.120923 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv"] Jul 02 19:01:56.124543 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:56.124521 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c532a-predictor-6988ddc98f-c4fgv"] Jul 02 19:01:57.235998 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:57.235969 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" path="/var/lib/kubelet/pods/d6dcad90-e414-454f-bf37-f2c6d826c0f1/volumes" Jul 02 19:01:59.036859 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:01:59.036817 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:02:00.103523 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:00.103489 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:02:00.104107 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:00.104080 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 02 19:02:00.949899 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:00.949861 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 02 19:02:04.036169 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:04.036126 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:02:04.036565 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:04.036239 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:02:09.036562 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:09.036525 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:02:10.104052 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:10.104012 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 02 19:02:10.949590 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:10.949559 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:02:14.035739 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:14.035698 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:02:19.036421 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:19.036382 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:02:20.104282 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:20.104246 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 02 19:02:22.043354 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.043320 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996"] Jul 02 19:02:22.043910 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.043895 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kube-rbac-proxy" Jul 02 19:02:22.043956 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.043912 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kube-rbac-proxy" Jul 02 19:02:22.043956 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.043931 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" Jul 02 19:02:22.043956 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.043937 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" Jul 02 19:02:22.044078 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.043995 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kube-rbac-proxy" Jul 02 19:02:22.044078 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.044007 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6dcad90-e414-454f-bf37-f2c6d826c0f1" containerName="kserve-container" Jul 02 19:02:22.047096 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.047080 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:22.048941 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.048918 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-208dc-kube-rbac-proxy-sar-config\"" Jul 02 19:02:22.049051 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.048990 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-208dc-serving-cert\"" Jul 02 19:02:22.052788 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.052766 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996"] Jul 02 19:02:22.107372 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.107344 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-proxy-tls\") pod \"switch-graph-208dc-64b8656f78-th996\" (UID: \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\") " pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:22.107473 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.107381 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-openshift-service-ca-bundle\") pod \"switch-graph-208dc-64b8656f78-th996\" (UID: \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\") " pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:22.208702 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.208667 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-proxy-tls\") pod \"switch-graph-208dc-64b8656f78-th996\" (UID: \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\") " pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:22.208860 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.208727 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-openshift-service-ca-bundle\") pod \"switch-graph-208dc-64b8656f78-th996\" (UID: \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\") " pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:22.208860 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:02:22.208817 2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-208dc-serving-cert: secret "switch-graph-208dc-serving-cert" not found Jul 02 19:02:22.208960 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:02:22.208887 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-proxy-tls podName:ff90b279-e314-4f0a-a1e3-5c715a4a7e28 nodeName:}" failed. No retries permitted until 2026-07-02 19:02:22.708868995 +0000 UTC m=+1899.995218820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-proxy-tls") pod "switch-graph-208dc-64b8656f78-th996" (UID: "ff90b279-e314-4f0a-a1e3-5c715a4a7e28") : secret "switch-graph-208dc-serving-cert" not found Jul 02 19:02:22.209293 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.209275 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-openshift-service-ca-bundle\") pod \"switch-graph-208dc-64b8656f78-th996\" (UID: \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\") " pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:22.224428 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.224403 2552 generic.go:356] "Generic (PLEG): container finished" podID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerID="df7e2524c87b87009792a8235777bd65fa016b651ad7a4f8d6f0eb87713b2762" exitCode=0 Jul 02 19:02:22.224555 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.224466 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" event={"ID":"61d3472c-ea65-43b2-beaf-387cdbb7ea44","Type":"ContainerDied","Data":"df7e2524c87b87009792a8235777bd65fa016b651ad7a4f8d6f0eb87713b2762"} Jul 02 19:02:22.698386 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.698363 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:02:22.714013 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.713987 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-proxy-tls\") pod \"switch-graph-208dc-64b8656f78-th996\" (UID: \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\") " pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:22.716490 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.716469 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-proxy-tls\") pod \"switch-graph-208dc-64b8656f78-th996\" (UID: \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\") " pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:22.814846 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.814818 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61d3472c-ea65-43b2-beaf-387cdbb7ea44-proxy-tls\") pod \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\" (UID: \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\") " Jul 02 19:02:22.815014 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.814899 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61d3472c-ea65-43b2-beaf-387cdbb7ea44-openshift-service-ca-bundle\") pod \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\" (UID: \"61d3472c-ea65-43b2-beaf-387cdbb7ea44\") " Jul 02 19:02:22.815260 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.815235 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d3472c-ea65-43b2-beaf-387cdbb7ea44-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "61d3472c-ea65-43b2-beaf-387cdbb7ea44" (UID: "61d3472c-ea65-43b2-beaf-387cdbb7ea44"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:02:22.816920 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.816884 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d3472c-ea65-43b2-beaf-387cdbb7ea44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61d3472c-ea65-43b2-beaf-387cdbb7ea44" (UID: "61d3472c-ea65-43b2-beaf-387cdbb7ea44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:02:22.915932 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.915864 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61d3472c-ea65-43b2-beaf-387cdbb7ea44-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:02:22.915932 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.915888 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61d3472c-ea65-43b2-beaf-387cdbb7ea44-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:02:22.958758 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:22.958735 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:23.085762 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:23.085737 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996"] Jul 02 19:02:23.088132 ip-10-0-130-23 kubenswrapper[2552]: W0702 19:02:23.088103 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff90b279_e314_4f0a_a1e3_5c715a4a7e28.slice/crio-23baed4c0413ffe49ee64f64f9971a15f9efb550343543eba0c557fe71775322 WatchSource:0}: Error finding container 23baed4c0413ffe49ee64f64f9971a15f9efb550343543eba0c557fe71775322: Status 404 returned error can't find the container with id 23baed4c0413ffe49ee64f64f9971a15f9efb550343543eba0c557fe71775322 Jul 02 19:02:23.232601 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:23.232570 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" Jul 02 19:02:23.235933 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:23.235910 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" event={"ID":"ff90b279-e314-4f0a-a1e3-5c715a4a7e28","Type":"ContainerStarted","Data":"3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650"} Jul 02 19:02:23.236061 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:23.235948 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" event={"ID":"ff90b279-e314-4f0a-a1e3-5c715a4a7e28","Type":"ContainerStarted","Data":"23baed4c0413ffe49ee64f64f9971a15f9efb550343543eba0c557fe71775322"} Jul 02 19:02:23.236061 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:23.235966 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb" event={"ID":"61d3472c-ea65-43b2-beaf-387cdbb7ea44","Type":"ContainerDied","Data":"6e1c241312a7040940c25df7aa64a291b30b3380a8b7161cf297be3671d16436"} Jul 02 19:02:23.236061 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:23.235989 2552 scope.go:117] "RemoveContainer" containerID="df7e2524c87b87009792a8235777bd65fa016b651ad7a4f8d6f0eb87713b2762" Jul 02 19:02:23.265463 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:23.265423 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" podStartSLOduration=1.265409347 podStartE2EDuration="1.265409347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:02:23.263327361 +0000 UTC m=+1900.549677215" watchObservedRunningTime="2026-07-02 19:02:23.265409347 +0000 UTC m=+1900.551759189" Jul 02 19:02:23.275310 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:23.275282 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb"] Jul 02 19:02:23.278288 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:23.278267 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c532a-76d7947565-k9nvb"] Jul 02 19:02:24.237105 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:24.237080 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:25.236198 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:25.236168 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" path="/var/lib/kubelet/pods/61d3472c-ea65-43b2-beaf-387cdbb7ea44/volumes" Jul 02 19:02:30.104295 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:30.104261 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 02 19:02:30.246849 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:30.246824 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:02:40.104901 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:40.104876 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:02:52.280673 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.280623 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh"] Jul 02 19:02:52.281211 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.281190 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" Jul 02 19:02:52.281303 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.281213 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" Jul 02 19:02:52.281356 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.281315 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d3472c-ea65-43b2-beaf-387cdbb7ea44" containerName="splitter-graph-c532a" Jul 02 19:02:52.284279 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.284251 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:02:52.286043 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.285990 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-c41f9-kube-rbac-proxy-sar-config\"" Jul 02 19:02:52.286155 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.286075 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-c41f9-serving-cert\"" Jul 02 19:02:52.291595 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.291564 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh"] Jul 02 19:02:52.467397 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.467361 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-openshift-service-ca-bundle\") pod \"splitter-graph-c41f9-5bc8d4797c-wprsh\" (UID: \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\") " pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:02:52.467579 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.467414 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-proxy-tls\") pod \"splitter-graph-c41f9-5bc8d4797c-wprsh\" (UID: \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\") " pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:02:52.568387 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.568299 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-openshift-service-ca-bundle\") pod \"splitter-graph-c41f9-5bc8d4797c-wprsh\" (UID: \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\") " pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:02:52.568387 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.568346 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-proxy-tls\") pod \"splitter-graph-c41f9-5bc8d4797c-wprsh\" (UID: \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\") " pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:02:52.568969 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.568945 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-openshift-service-ca-bundle\") pod \"splitter-graph-c41f9-5bc8d4797c-wprsh\" (UID: \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\") " pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:02:52.570718 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.570695 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-proxy-tls\") pod \"splitter-graph-c41f9-5bc8d4797c-wprsh\" (UID: \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\") " pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:02:52.600294 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.600270 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:02:52.721495 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:52.721469 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh"] Jul 02 19:02:52.723780 ip-10-0-130-23 kubenswrapper[2552]: W0702 19:02:52.723751 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d94ec8_ec00_4692_a1e4_03fdb47f5ca8.slice/crio-5b98c54f3ab15ee497e4d44c37b55f82035c526500804466507f5c999e76c36f WatchSource:0}: Error finding container 5b98c54f3ab15ee497e4d44c37b55f82035c526500804466507f5c999e76c36f: Status 404 returned error can't find the container with id 5b98c54f3ab15ee497e4d44c37b55f82035c526500804466507f5c999e76c36f Jul 02 19:02:53.362601 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:53.362569 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" event={"ID":"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8","Type":"ContainerStarted","Data":"68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a"} Jul 02 19:02:53.363083 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:53.362625 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" event={"ID":"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8","Type":"ContainerStarted","Data":"5b98c54f3ab15ee497e4d44c37b55f82035c526500804466507f5c999e76c36f"} Jul 02 19:02:53.363083 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:53.362708 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:02:53.378482 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:53.378436 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" podStartSLOduration=1.378420944 podStartE2EDuration="1.378420944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:02:53.376013392 +0000 UTC m=+1930.662363235" watchObservedRunningTime="2026-07-02 19:02:53.378420944 +0000 UTC m=+1930.664770788" Jul 02 19:02:59.372842 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:02:59.372811 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:11:06.949754 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:06.949670 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh"] Jul 02 19:11:06.950331 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:06.949909 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" containerID="cri-o://68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a" gracePeriod=30 Jul 02 19:11:07.076018 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:07.075983 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt"] Jul 02 19:11:07.076316 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:07.076266 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" containerID="cri-o://7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d" gracePeriod=30 Jul 02 19:11:07.076603 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:07.076561 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kube-rbac-proxy" containerID="cri-o://ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85" gracePeriod=30 Jul 02 19:11:07.599779 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:07.599748 2552 generic.go:356] "Generic (PLEG): container finished" podID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerID="ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85" exitCode=2 Jul 02 19:11:07.599958 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:07.599797 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" event={"ID":"29f316cb-9604-4d0e-b80e-a5c151e307ae","Type":"ContainerDied","Data":"ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85"} Jul 02 19:11:09.371242 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:09.371201 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:11:10.131482 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.131459 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:11:10.200003 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.199966 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2vtd\" (UniqueName: \"kubernetes.io/projected/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-api-access-d2vtd\") pod \"29f316cb-9604-4d0e-b80e-a5c151e307ae\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " Jul 02 19:11:10.200170 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.200092 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f316cb-9604-4d0e-b80e-a5c151e307ae-proxy-tls\") pod \"29f316cb-9604-4d0e-b80e-a5c151e307ae\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " Jul 02 19:11:10.200170 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.200131 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-rbac-proxy-sar-config\") pod \"29f316cb-9604-4d0e-b80e-a5c151e307ae\" (UID: \"29f316cb-9604-4d0e-b80e-a5c151e307ae\") " Jul 02 19:11:10.200531 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.200500 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29f316cb-9604-4d0e-b80e-a5c151e307ae" (UID: "29f316cb-9604-4d0e-b80e-a5c151e307ae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:10.202018 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.201995 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-api-access-d2vtd" (OuterVolumeSpecName: "kube-api-access-d2vtd") pod "29f316cb-9604-4d0e-b80e-a5c151e307ae" (UID: "29f316cb-9604-4d0e-b80e-a5c151e307ae"). InnerVolumeSpecName "kube-api-access-d2vtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:11:10.202091 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.202061 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f316cb-9604-4d0e-b80e-a5c151e307ae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29f316cb-9604-4d0e-b80e-a5c151e307ae" (UID: "29f316cb-9604-4d0e-b80e-a5c151e307ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:11:10.301046 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.300964 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f316cb-9604-4d0e-b80e-a5c151e307ae-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:11:10.301046 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.300993 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:11:10.301046 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.301002 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vtd\" (UniqueName: \"kubernetes.io/projected/29f316cb-9604-4d0e-b80e-a5c151e307ae-kube-api-access-d2vtd\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:11:10.614575 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.614475 2552 generic.go:356] "Generic (PLEG): container finished" podID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerID="7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d" exitCode=0 Jul 02 19:11:10.614575 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.614529 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" Jul 02 19:11:10.614575 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.614541 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" event={"ID":"29f316cb-9604-4d0e-b80e-a5c151e307ae","Type":"ContainerDied","Data":"7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d"} Jul 02 19:11:10.615189 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.614583 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" event={"ID":"29f316cb-9604-4d0e-b80e-a5c151e307ae","Type":"ContainerDied","Data":"7c9a4b38234dadf6d301e68c3ae08b9db8246be002d78fefed4d688f86fc498f"} Jul 02 19:11:10.615189 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.614608 2552 scope.go:117] "RemoveContainer" containerID="ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85" Jul 02 19:11:10.624269 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.624249 2552 scope.go:117] "RemoveContainer" containerID="7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d" Jul 02 19:11:10.632017 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.631996 2552 scope.go:117] "RemoveContainer" containerID="ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85" Jul 02 19:11:10.632294 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:11:10.632271 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85\": container with ID starting with ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85 not found: ID does not exist" containerID="ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85" Jul 02 19:11:10.632368 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.632303 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85"} err="failed to get container status \"ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85\": rpc error: code = NotFound desc = could not find container \"ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85\": container with ID starting with ddcb345866a38571575fdef3f17ebb762c5ef114e974b88a5b2222b1ae8c1a85 not found: ID does not exist" Jul 02 19:11:10.632368 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.632321 2552 scope.go:117] "RemoveContainer" containerID="7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d" Jul 02 19:11:10.632563 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:11:10.632542 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d\": container with ID starting with 7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d not found: ID does not exist" containerID="7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d" Jul 02 19:11:10.632601 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.632571 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d"} err="failed to get container status \"7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d\": rpc error: code = NotFound desc = could not find container \"7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d\": container with ID starting with 7d4c97ccc86e9d9187624898860120436804319575fbb199969ce4f8d4931c7d not found: ID does not exist" Jul 02 19:11:10.636336 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.636314 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt"] Jul 02 19:11:10.641221 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:10.641200 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt"] Jul 02 19:11:10.643602 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:11:10.643582 2552 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f316cb_9604_4d0e_b80e_a5c151e307ae.slice\": RecentStats: unable to find data in memory cache]" Jul 02 19:11:11.099920 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:11.099877 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": context deadline exceeded" Jul 02 19:11:11.105142 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:11.105114 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c41f9-predictor-bf7df5ddc-9gcbt" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: i/o timeout" Jul 02 19:11:11.237886 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:11.237853 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" path="/var/lib/kubelet/pods/29f316cb-9604-4d0e-b80e-a5c151e307ae/volumes" Jul 02 19:11:14.371326 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:14.371291 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:11:19.371623 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:19.371584 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:11:19.372029 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:19.371700 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:11:24.371177 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:24.371134 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:11:29.371440 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:29.371395 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:11:34.371384 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:34.371348 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:11:37.107794 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.107772 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:11:37.225971 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.225870 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-proxy-tls\") pod \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\" (UID: \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\") " Jul 02 19:11:37.225971 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.225929 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-openshift-service-ca-bundle\") pod \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\" (UID: \"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8\") " Jul 02 19:11:37.226359 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.226329 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" (UID: "07d94ec8-ec00-4692-a1e4-03fdb47f5ca8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:37.227974 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.227954 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" (UID: "07d94ec8-ec00-4692-a1e4-03fdb47f5ca8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:11:37.327164 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.327136 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:11:37.327164 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.327163 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:11:37.737720 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.737684 2552 generic.go:356] "Generic (PLEG): container finished" podID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerID="68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a" exitCode=0 Jul 02 19:11:37.737916 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.737733 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" Jul 02 19:11:37.737916 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.737765 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" event={"ID":"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8","Type":"ContainerDied","Data":"68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a"} Jul 02 19:11:37.737916 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.737820 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh" event={"ID":"07d94ec8-ec00-4692-a1e4-03fdb47f5ca8","Type":"ContainerDied","Data":"5b98c54f3ab15ee497e4d44c37b55f82035c526500804466507f5c999e76c36f"} Jul 02 19:11:37.737916 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.737843 2552 scope.go:117] "RemoveContainer" containerID="68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a" Jul 02 19:11:37.746183 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.746167 2552 scope.go:117] "RemoveContainer" containerID="68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a" Jul 02 19:11:37.746450 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:11:37.746434 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a\": container with ID starting with 68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a not found: ID does not exist" containerID="68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a" Jul 02 19:11:37.746514 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.746455 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a"} err="failed to get container status \"68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a\": rpc error: code = NotFound desc = could not find container \"68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a\": container with ID starting with 68ad5fcbe9a0fda5b5735b049fe0ba077c7b2403ec54fa04ad82fdfaadf8235a not found: ID does not exist" Jul 02 19:11:37.752949 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.752926 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh"] Jul 02 19:11:37.756784 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:37.756763 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-c41f9-5bc8d4797c-wprsh"] Jul 02 19:11:39.236737 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:11:39.236706 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" path="/var/lib/kubelet/pods/07d94ec8-ec00-4692-a1e4-03fdb47f5ca8/volumes" Jul 02 19:18:41.437040 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:41.436954 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996"] Jul 02 19:18:41.439522 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:41.437198 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" containerID="cri-o://3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650" gracePeriod=30 Jul 02 19:18:41.576141 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:41.576106 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7"] Jul 02 19:18:41.576374 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:41.576346 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" containerID="cri-o://7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f" gracePeriod=30 Jul 02 19:18:41.576456 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:41.576379 2552 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kube-rbac-proxy" containerID="cri-o://10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1" gracePeriod=30 Jul 02 19:18:42.639985 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:42.639942 2552 generic.go:356] "Generic (PLEG): container finished" podID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerID="10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1" exitCode=2 Jul 02 19:18:42.640355 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:42.640015 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" event={"ID":"704b39e9-fa02-4592-8f70-01d5bd2bacbf","Type":"ContainerDied","Data":"10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1"} Jul 02 19:18:44.621688 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.621665 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:18:44.649535 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.649506 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b39e9-fa02-4592-8f70-01d5bd2bacbf-proxy-tls\") pod \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " Jul 02 19:18:44.649712 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.649599 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-rbac-proxy-sar-config\") pod \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " Jul 02 19:18:44.649712 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.649621 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhqjk\" (UniqueName: \"kubernetes.io/projected/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-api-access-jhqjk\") pod \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\" (UID: \"704b39e9-fa02-4592-8f70-01d5bd2bacbf\") " Jul 02 19:18:44.649995 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.649960 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "704b39e9-fa02-4592-8f70-01d5bd2bacbf" (UID: "704b39e9-fa02-4592-8f70-01d5bd2bacbf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:18:44.651223 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.651194 2552 generic.go:356] "Generic (PLEG): container finished" podID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerID="7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f" exitCode=0 Jul 02 19:18:44.651323 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.651262 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" Jul 02 19:18:44.651323 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.651277 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" event={"ID":"704b39e9-fa02-4592-8f70-01d5bd2bacbf","Type":"ContainerDied","Data":"7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f"} Jul 02 19:18:44.651403 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.651378 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7" event={"ID":"704b39e9-fa02-4592-8f70-01d5bd2bacbf","Type":"ContainerDied","Data":"e88a401ffcfbf32faeaecfe30e807d79027a4909af00274543df1dc85ef6b2e5"} Jul 02 19:18:44.651448 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.651417 2552 scope.go:117] "RemoveContainer" containerID="10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1" Jul 02 19:18:44.651914 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.651894 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704b39e9-fa02-4592-8f70-01d5bd2bacbf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "704b39e9-fa02-4592-8f70-01d5bd2bacbf" (UID: "704b39e9-fa02-4592-8f70-01d5bd2bacbf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:18:44.652389 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.652363 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-api-access-jhqjk" (OuterVolumeSpecName: "kube-api-access-jhqjk") pod "704b39e9-fa02-4592-8f70-01d5bd2bacbf" (UID: "704b39e9-fa02-4592-8f70-01d5bd2bacbf"). InnerVolumeSpecName "kube-api-access-jhqjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:18:44.673788 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.673761 2552 scope.go:117] "RemoveContainer" containerID="7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f" Jul 02 19:18:44.680726 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.680708 2552 scope.go:117] "RemoveContainer" containerID="10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1" Jul 02 19:18:44.680978 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:18:44.680959 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1\": container with ID starting with 10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1 not found: ID does not exist" containerID="10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1" Jul 02 19:18:44.681030 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.680989 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1"} err="failed to get container status \"10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1\": rpc error: code = NotFound desc = could not find container \"10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1\": container with ID starting with 10f45a50c8d3dc5e03203c9122a461150a99ab1dceef1dec1d921fbf407690a1 not found: ID does not exist" Jul 02 19:18:44.681030 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.681006 2552 scope.go:117] "RemoveContainer" containerID="7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f" Jul 02 19:18:44.681227 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:18:44.681210 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f\": container with ID starting with 7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f not found: ID does not exist" containerID="7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f" Jul 02 19:18:44.681268 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.681232 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f"} err="failed to get container status \"7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f\": rpc error: code = NotFound desc = could not find container \"7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f\": container with ID starting with 7b4fab8c88a15b73c297f53bb67ba422d27c27e3080d83904604174679a3408f not found: ID does not exist" Jul 02 19:18:44.750992 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.750935 2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:18:44.750992 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.750954 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqjk\" (UniqueName: \"kubernetes.io/projected/704b39e9-fa02-4592-8f70-01d5bd2bacbf-kube-api-access-jhqjk\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:18:44.750992 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.750964 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b39e9-fa02-4592-8f70-01d5bd2bacbf-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:18:44.970560 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.970537 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7"] Jul 02 19:18:44.973877 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:44.973857 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-208dc-predictor-7b67d8c5b7-lmdz7"] Jul 02 19:18:45.237050 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:45.237021 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" path="/var/lib/kubelet/pods/704b39e9-fa02-4592-8f70-01d5bd2bacbf/volumes" Jul 02 19:18:45.244726 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:45.244699 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:18:50.244913 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:50.244874 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:18:55.244965 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:55.244926 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:18:55.245428 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:55.245044 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:18:58.413362 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:58.413332 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:18:59.219516 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:18:59.219483 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:00.032359 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:00.032327 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:00.245100 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:00.245066 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:19:00.813866 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:00.813823 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:01.604048 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:01.604022 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:02.384409 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:02.384379 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:03.188288 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:03.188262 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:03.980323 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:03.980295 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:04.777478 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:04.777445 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:05.245029 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:05.244993 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:19:05.599602 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:05.599517 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:06.383979 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:06.383944 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:07.180248 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:07.180217 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:07.977514 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:07.977484 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:08.832532 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:08.832500 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-208dc-64b8656f78-th996_ff90b279-e314-4f0a-a1e3-5c715a4a7e28/switch-graph-208dc/0.log" Jul 02 19:19:10.244854 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:10.244808 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:19:11.602567 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.602539 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:19:11.677061 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.677027 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-proxy-tls\") pod \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\" (UID: \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\") " Jul 02 19:19:11.677254 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.677086 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-openshift-service-ca-bundle\") pod \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\" (UID: \"ff90b279-e314-4f0a-a1e3-5c715a4a7e28\") " Jul 02 19:19:11.677470 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.677446 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ff90b279-e314-4f0a-a1e3-5c715a4a7e28" (UID: "ff90b279-e314-4f0a-a1e3-5c715a4a7e28"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:19:11.679107 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.679083 2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff90b279-e314-4f0a-a1e3-5c715a4a7e28" (UID: "ff90b279-e314-4f0a-a1e3-5c715a4a7e28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:19:11.777058 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.776966 2552 generic.go:356] "Generic (PLEG): container finished" podID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerID="3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650" exitCode=0 Jul 02 19:19:11.777058 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.777012 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" Jul 02 19:19:11.777058 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.777029 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" event={"ID":"ff90b279-e314-4f0a-a1e3-5c715a4a7e28","Type":"ContainerDied","Data":"3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650"} Jul 02 19:19:11.777318 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.777072 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996" event={"ID":"ff90b279-e314-4f0a-a1e3-5c715a4a7e28","Type":"ContainerDied","Data":"23baed4c0413ffe49ee64f64f9971a15f9efb550343543eba0c557fe71775322"} Jul 02 19:19:11.777318 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.777096 2552 scope.go:117] "RemoveContainer" containerID="3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650" Jul 02 19:19:11.777589 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.777569 2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-proxy-tls\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:19:11.777658 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.777596 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff90b279-e314-4f0a-a1e3-5c715a4a7e28-openshift-service-ca-bundle\") on node \"ip-10-0-130-23.ec2.internal\" DevicePath \"\"" Jul 02 19:19:11.786216 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.786188 2552 scope.go:117] "RemoveContainer" containerID="3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650" Jul 02 19:19:11.786771 ip-10-0-130-23 kubenswrapper[2552]: E0702 19:19:11.786703 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650\": container with ID starting with 3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650 not found: ID does not exist" containerID="3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650" Jul 02 19:19:11.786771 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.786743 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650"} err="failed to get container status \"3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650\": rpc error: code = NotFound desc = could not find container \"3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650\": container with ID starting with 3443899bdbe0b03b8801478bfc1c3bf47828f05b824bd4ba7a26410ff8494650 not found: ID does not exist" Jul 02 19:19:11.803948 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.803911 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996"] Jul 02 19:19:11.806091 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:11.806059 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-208dc-64b8656f78-th996"] Jul 02 19:19:13.069146 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069107 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xqmdr/must-gather-6l9sc"] Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069484 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069495 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069508 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069514 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069521 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kube-rbac-proxy" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069527 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kube-rbac-proxy" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069536 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069542 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069549 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069554 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069563 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kube-rbac-proxy" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069568 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kube-rbac-proxy" Jul 02 19:19:13.069621 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069629 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kube-rbac-proxy" Jul 02 19:19:13.070093 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069652 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kserve-container" Jul 02 19:19:13.070093 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069660 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="29f316cb-9604-4d0e-b80e-a5c151e307ae" containerName="kube-rbac-proxy" Jul 02 19:19:13.070093 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069666 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" containerName="switch-graph-208dc" Jul 02 19:19:13.070093 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069671 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="704b39e9-fa02-4592-8f70-01d5bd2bacbf" containerName="kserve-container" Jul 02 19:19:13.070093 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.069677 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d94ec8-ec00-4692-a1e4-03fdb47f5ca8" containerName="splitter-graph-c41f9" Jul 02 19:19:13.073979 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.073958 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xqmdr/must-gather-6l9sc" Jul 02 19:19:13.076115 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.076082 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xqmdr\"/\"openshift-service-ca.crt\"" Jul 02 19:19:13.076512 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.076485 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xqmdr\"/\"kube-root-ca.crt\"" Jul 02 19:19:13.076676 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.076605 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xqmdr\"/\"default-dockercfg-k4zn2\"" Jul 02 19:19:13.078776 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.078753 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xqmdr/must-gather-6l9sc"] Jul 02 19:19:13.189733 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.189691 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b-must-gather-output\") pod \"must-gather-6l9sc\" (UID: \"d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b\") " pod="openshift-must-gather-xqmdr/must-gather-6l9sc" Jul 02 19:19:13.189919 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.189760 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6l64\" (UniqueName: \"kubernetes.io/projected/d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b-kube-api-access-r6l64\") pod \"must-gather-6l9sc\" (UID: \"d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b\") " pod="openshift-must-gather-xqmdr/must-gather-6l9sc" Jul 02 19:19:13.236843 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.236812 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff90b279-e314-4f0a-a1e3-5c715a4a7e28" path="/var/lib/kubelet/pods/ff90b279-e314-4f0a-a1e3-5c715a4a7e28/volumes" Jul 02 19:19:13.291212 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.291164 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b-must-gather-output\") pod \"must-gather-6l9sc\" (UID: \"d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b\") " pod="openshift-must-gather-xqmdr/must-gather-6l9sc" Jul 02 19:19:13.291355 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.291252 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6l64\" (UniqueName: \"kubernetes.io/projected/d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b-kube-api-access-r6l64\") pod \"must-gather-6l9sc\" (UID: \"d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b\") " pod="openshift-must-gather-xqmdr/must-gather-6l9sc" Jul 02 19:19:13.291545 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.291528 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b-must-gather-output\") pod \"must-gather-6l9sc\" (UID: \"d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b\") " pod="openshift-must-gather-xqmdr/must-gather-6l9sc" Jul 02 19:19:13.299309 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.299274 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6l64\" (UniqueName: \"kubernetes.io/projected/d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b-kube-api-access-r6l64\") pod \"must-gather-6l9sc\" (UID: \"d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b\") " pod="openshift-must-gather-xqmdr/must-gather-6l9sc" Jul 02 19:19:13.383754 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.383664 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xqmdr/must-gather-6l9sc" Jul 02 19:19:13.515481 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.515374 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xqmdr/must-gather-6l9sc"] Jul 02 19:19:13.518089 ip-10-0-130-23 kubenswrapper[2552]: W0702 19:19:13.518062 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd64162d0_cc48_4f0b_90d2_8aea6dc0fe8b.slice/crio-1c0e65aec95899f7ebf7eecb3bdda434f68c14bc933df2ad71d365be3ac5a400 WatchSource:0}: Error finding container 1c0e65aec95899f7ebf7eecb3bdda434f68c14bc933df2ad71d365be3ac5a400: Status 404 returned error can't find the container with id 1c0e65aec95899f7ebf7eecb3bdda434f68c14bc933df2ad71d365be3ac5a400 Jul 02 19:19:13.519852 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.519833 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:19:13.787912 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:13.787876 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xqmdr/must-gather-6l9sc" event={"ID":"d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b","Type":"ContainerStarted","Data":"1c0e65aec95899f7ebf7eecb3bdda434f68c14bc933df2ad71d365be3ac5a400"} Jul 02 19:19:14.796278 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:14.796235 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xqmdr/must-gather-6l9sc" event={"ID":"d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b","Type":"ContainerStarted","Data":"a552c6a9d9aa65850137e983328bd23e73a09caf3d79195c6f07c5076185e55b"} Jul 02 19:19:14.796278 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:14.796276 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xqmdr/must-gather-6l9sc" event={"ID":"d64162d0-cc48-4f0b-90d2-8aea6dc0fe8b","Type":"ContainerStarted","Data":"2b7fe9e1474358924eeed4126919c98331e74596b644de6e325ab39e376ba9ac"} Jul 02 19:19:15.701743 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:15.701716 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hjqpc_d103adac-ed3a-485a-90c7-9713adec7aba/global-pull-secret-syncer/0.log" Jul 02 19:19:15.917090 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:15.917055 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qkjmb_d68dc90b-8983-41f7-a1d7-6c23581adfbe/konnectivity-agent/0.log" Jul 02 19:19:15.945140 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:15.945103 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-23.ec2.internal_26e94d30503c9daf0c2ddc17f9303308/haproxy/0.log" Jul 02 19:19:20.189690 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.189660 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6cae0007-99f0-4e1d-a521-97ce445173d0/alertmanager/0.log" Jul 02 19:19:20.213215 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.213123 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6cae0007-99f0-4e1d-a521-97ce445173d0/config-reloader/0.log" Jul 02 19:19:20.237544 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.237507 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6cae0007-99f0-4e1d-a521-97ce445173d0/kube-rbac-proxy-web/0.log" Jul 02 19:19:20.262129 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.262098 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6cae0007-99f0-4e1d-a521-97ce445173d0/kube-rbac-proxy/0.log" Jul 02 19:19:20.287835 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.287807 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6cae0007-99f0-4e1d-a521-97ce445173d0/kube-rbac-proxy-metric/0.log" Jul 02 19:19:20.314313 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.314281 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6cae0007-99f0-4e1d-a521-97ce445173d0/prom-label-proxy/0.log" Jul 02 19:19:20.339938 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.339912 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6cae0007-99f0-4e1d-a521-97ce445173d0/init-config-reloader/0.log" Jul 02 19:19:20.414430 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.414370 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-twjc7_f7c990c6-db5c-4ac4-9599-3a734e3360fe/kube-state-metrics/0.log" Jul 02 19:19:20.437030 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.436976 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-twjc7_f7c990c6-db5c-4ac4-9599-3a734e3360fe/kube-rbac-proxy-main/0.log" Jul 02 19:19:20.465427 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.465340 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-twjc7_f7c990c6-db5c-4ac4-9599-3a734e3360fe/kube-rbac-proxy-self/0.log" Jul 02 19:19:20.541004 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.540912 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-654dcbc7d6-69zxg_41c322b7-6bea-4cea-8cb5-fe48d49d4d32/metrics-server/0.log" Jul 02 19:19:20.578719 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.578670 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-mhwpt_1ad4eb8b-165f-4a95-ab78-1b7c9986bf46/monitoring-plugin/0.log" Jul 02 19:19:20.698471 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.698432 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cwwxs_8e0002e3-0a86-4c4e-bf05-cef319fcd37b/node-exporter/0.log" Jul 02 19:19:20.722688 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.722595 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cwwxs_8e0002e3-0a86-4c4e-bf05-cef319fcd37b/kube-rbac-proxy/0.log" Jul 02 19:19:20.749189 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.749156 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cwwxs_8e0002e3-0a86-4c4e-bf05-cef319fcd37b/init-textfile/0.log" Jul 02 19:19:20.866942 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.866909 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-dl759_248d42be-542d-44c5-a37c-901bf74745d4/kube-rbac-proxy-main/0.log" Jul 02 19:19:20.890540 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.890503 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-dl759_248d42be-542d-44c5-a37c-901bf74745d4/kube-rbac-proxy-self/0.log" Jul 02 19:19:20.915022 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:20.914990 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-dl759_248d42be-542d-44c5-a37c-901bf74745d4/openshift-state-metrics/0.log" Jul 02 19:19:21.131555 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:21.131466 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-j2z8q_0a04623b-23c8-43d9-b06e-4fa9507d4883/prometheus-operator/0.log" Jul 02 19:19:21.151351 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:21.151310 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-j2z8q_0a04623b-23c8-43d9-b06e-4fa9507d4883/kube-rbac-proxy/0.log" Jul 02 19:19:21.286784 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:21.286750 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c9f6559d8-8gg8n_b172d8bd-adf9-412f-be4a-f47f149ffdfb/thanos-query/0.log" Jul 02 19:19:21.313170 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:21.313139 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c9f6559d8-8gg8n_b172d8bd-adf9-412f-be4a-f47f149ffdfb/kube-rbac-proxy-web/0.log" Jul 02 19:19:21.336906 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:21.336880 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c9f6559d8-8gg8n_b172d8bd-adf9-412f-be4a-f47f149ffdfb/kube-rbac-proxy/0.log" Jul 02 19:19:21.359405 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:21.359378 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c9f6559d8-8gg8n_b172d8bd-adf9-412f-be4a-f47f149ffdfb/prom-label-proxy/0.log" Jul 02 19:19:21.384219 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:21.384118 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c9f6559d8-8gg8n_b172d8bd-adf9-412f-be4a-f47f149ffdfb/kube-rbac-proxy-rules/0.log" Jul 02 19:19:21.410763 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:21.410737 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c9f6559d8-8gg8n_b172d8bd-adf9-412f-be4a-f47f149ffdfb/kube-rbac-proxy-metrics/0.log" Jul 02 19:19:23.629307 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:23.629281 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b858d7ddb-qmprg_95ba0dee-2108-4859-a2c5-ab6ed665840b/console/0.log" Jul 02 19:19:24.339499 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.339432 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xqmdr/must-gather-6l9sc" podStartSLOduration=10.575301662 podStartE2EDuration="11.339406305s" totalImagesPullingTime="764.104643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:19:14.810429854 +0000 UTC m=+2912.096779698" watchObservedRunningTime="2026-07-02 19:19:24.339406305 +0000 UTC m=+2921.625756151" Jul 02 19:19:24.341453 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.341420 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t"] Jul 02 19:19:24.347683 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.347661 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.354266 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.354234 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t"] Jul 02 19:19:24.501846 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.501814 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-podres\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.502026 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.501865 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-proc\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.502026 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.501924 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-sys\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.502026 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.502002 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-lib-modules\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.502187 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.502103 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkql\" (UniqueName: \"kubernetes.io/projected/2987402c-22d3-4316-84b7-6e76a49a0a78-kube-api-access-5pkql\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.603132 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.603041 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkql\" (UniqueName: \"kubernetes.io/projected/2987402c-22d3-4316-84b7-6e76a49a0a78-kube-api-access-5pkql\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.603132 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.603111 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-podres\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.603456 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.603140 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-proc\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.603456 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.603157 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-sys\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.603456 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.603181 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-lib-modules\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.603456 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.603248 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-proc\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.603456 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.603266 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-sys\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.603456 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.603287 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-podres\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.603456 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.603308 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2987402c-22d3-4316-84b7-6e76a49a0a78-lib-modules\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.611226 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.611197 2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkql\" (UniqueName: \"kubernetes.io/projected/2987402c-22d3-4316-84b7-6e76a49a0a78-kube-api-access-5pkql\") pod \"perf-node-gather-daemonset-2nq9t\" (UID: \"2987402c-22d3-4316-84b7-6e76a49a0a78\") " pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.660942 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.660904 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:24.844987 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.844947 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t"] Jul 02 19:19:24.848277 ip-10-0-130-23 kubenswrapper[2552]: W0702 19:19:24.848249 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2987402c_22d3_4316_84b7_6e76a49a0a78.slice/crio-e17f06d7a103f42170d364a4e0d40e0838327ab5deb6f4dd5a12aae7ca7952f4 WatchSource:0}: Error finding container e17f06d7a103f42170d364a4e0d40e0838327ab5deb6f4dd5a12aae7ca7952f4: Status 404 returned error can't find the container with id e17f06d7a103f42170d364a4e0d40e0838327ab5deb6f4dd5a12aae7ca7952f4 Jul 02 19:19:24.858219 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:24.858191 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" event={"ID":"2987402c-22d3-4316-84b7-6e76a49a0a78","Type":"ContainerStarted","Data":"e17f06d7a103f42170d364a4e0d40e0838327ab5deb6f4dd5a12aae7ca7952f4"} Jul 02 19:19:25.093824 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:25.093798 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4q74j_3de88ed6-612b-4cc2-b74d-648f18596a79/dns/0.log" Jul 02 19:19:25.119268 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:25.119202 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4q74j_3de88ed6-612b-4cc2-b74d-648f18596a79/kube-rbac-proxy/0.log" Jul 02 19:19:25.294060 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:25.294035 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hcdmt_7560ae93-bfc6-4b22-acde-054add93fe8d/dns-node-resolver/0.log" Jul 02 19:19:25.805256 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:25.805228 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gh2rn_c9f69688-0465-473e-bcab-909e7800ee07/node-ca/0.log" Jul 02 19:19:25.866696 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:25.866653 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" event={"ID":"2987402c-22d3-4316-84b7-6e76a49a0a78","Type":"ContainerStarted","Data":"cfd554728f605d098a7a83eb5d93af260b016e90a6a82254d5ba93873db86cc2"} Jul 02 19:19:25.866893 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:25.866834 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:25.884136 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:25.884073 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" podStartSLOduration=1.88405293 podStartE2EDuration="1.88405293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:19:25.881815758 +0000 UTC m=+2923.168165602" watchObservedRunningTime="2026-07-02 19:19:25.88405293 +0000 UTC m=+2923.170402774" Jul 02 19:19:26.980624 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:26.980586 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8gdgj_6cce07b1-d3ba-4353-963f-b3e9f0cf98a4/serve-healthcheck-canary/0.log" Jul 02 19:19:27.390435 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:27.390347 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mmrr_2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e/kube-rbac-proxy/0.log" Jul 02 19:19:27.413736 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:27.413706 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mmrr_2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e/exporter/0.log" Jul 02 19:19:27.435343 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:27.435319 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mmrr_2c4f85cb-78f7-4bcd-9993-8bfa039f7e7e/extractor/0.log" Jul 02 19:19:29.583450 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:29.583404 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7cd6c584bf-2xwxb_d94d3095-e3e6-472f-b7ea-87ef50a87b57/manager/0.log" Jul 02 19:19:29.631801 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:29.631767 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-l4dmh_8ed463a4-8399-44b5-b2e1-8d390dd194f0/server/0.log" Jul 02 19:19:29.745028 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:29.744989 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-nvr8r_b2246ded-1c56-4a1f-8348-86ff9c3941d2/manager/0.log" Jul 02 19:19:31.880609 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:31.880581 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xqmdr/perf-node-gather-daemonset-2nq9t" Jul 02 19:19:35.068452 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.068418 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7x7hr_474dc2d4-8565-47f3-8ebc-dfdf419e11b6/kube-multus/0.log" Jul 02 19:19:35.438127 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.438101 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r52rg_3016b638-7d5e-49c8-9a84-034ff4ab17f7/kube-multus-additional-cni-plugins/0.log" Jul 02 19:19:35.461521 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.461494 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r52rg_3016b638-7d5e-49c8-9a84-034ff4ab17f7/egress-router-binary-copy/0.log" Jul 02 19:19:35.487528 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.487494 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r52rg_3016b638-7d5e-49c8-9a84-034ff4ab17f7/cni-plugins/0.log" Jul 02 19:19:35.511458 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.511430 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r52rg_3016b638-7d5e-49c8-9a84-034ff4ab17f7/bond-cni-plugin/0.log" Jul 02 19:19:35.535045 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.535018 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r52rg_3016b638-7d5e-49c8-9a84-034ff4ab17f7/routeoverride-cni/0.log" Jul 02 19:19:35.559785 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.559618 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r52rg_3016b638-7d5e-49c8-9a84-034ff4ab17f7/whereabouts-cni-bincopy/0.log" Jul 02 19:19:35.582904 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.582838 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r52rg_3016b638-7d5e-49c8-9a84-034ff4ab17f7/whereabouts-cni/0.log" Jul 02 19:19:35.764625 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.764555 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qc7bk_d1038ec5-e712-43d1-bf5e-8370bcefd309/network-metrics-daemon/0.log" Jul 02 19:19:35.786191 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:35.786165 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qc7bk_d1038ec5-e712-43d1-bf5e-8370bcefd309/kube-rbac-proxy/0.log" Jul 02 19:19:36.494604 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:36.494571 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-59qz5_11a0b4e9-c404-4cf5-b70d-8736829d5821/ovn-controller/0.log" Jul 02 19:19:36.514797 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:36.514775 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-59qz5_11a0b4e9-c404-4cf5-b70d-8736829d5821/ovn-acl-logging/0.log" Jul 02 19:19:36.538448 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:36.538420 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-59qz5_11a0b4e9-c404-4cf5-b70d-8736829d5821/kube-rbac-proxy-node/0.log" Jul 02 19:19:36.565819 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:36.565778 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-59qz5_11a0b4e9-c404-4cf5-b70d-8736829d5821/kube-rbac-proxy-ovn-metrics/0.log" Jul 02 19:19:36.589232 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:36.589206 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-59qz5_11a0b4e9-c404-4cf5-b70d-8736829d5821/northd/0.log" Jul 02 19:19:36.613525 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:36.613486 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-59qz5_11a0b4e9-c404-4cf5-b70d-8736829d5821/nbdb/0.log" Jul 02 19:19:36.638230 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:36.638207 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-59qz5_11a0b4e9-c404-4cf5-b70d-8736829d5821/sbdb/0.log" Jul 02 19:19:36.692011 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:36.691970 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-59qz5_11a0b4e9-c404-4cf5-b70d-8736829d5821/ovnkube-controller/0.log" Jul 02 19:19:38.194935 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:38.194901 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-n98x8_91c774ed-1026-4a13-91ba-61cb58ae1bca/network-check-target-container/0.log" Jul 02 19:19:39.093594 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:39.093568 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jmq4v_cc54c34b-e67c-4e6a-bd82-a00e09e2d94e/iptables-alerter/0.log" Jul 02 19:19:39.727509 ip-10-0-130-23 kubenswrapper[2552]: I0702 19:19:39.727481 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-p4w84_c4de0e2a-fa13-4ffe-8835-9545f4f925af/tuned/0.log"