Jul 24 16:06:19.135481 ip-10-0-134-103 systemd[1]: Starting Kubernetes Kubelet... Jul 24 16:06:19.503278 ip-10-0-134-103 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 16:06:19.503278 ip-10-0-134-103 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 16:06:19.503278 ip-10-0-134-103 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 16:06:19.503278 ip-10-0-134-103 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 16:06:19.503278 ip-10-0-134-103 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 16:06:19.505136 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.505052 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 16:06:19.511621 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511590 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 16:06:19.511621 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511616 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 16:06:19.511621 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511620 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 16:06:19.511621 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511623 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 16:06:19.511621 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511626 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 16:06:19.511621 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511630 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511633 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511636 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511639 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511642 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511644 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511647 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511650 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511652 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511655 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511658 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511662 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511667 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511670 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511673 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511676 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511678 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511680 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511683 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 16:06:19.511842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511686 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511688 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511691 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511694 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511696 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511698 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511701 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511704 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511707 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511709 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511712 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511714 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511716 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511719 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511721 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511724 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511727 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511730 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511733 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 16:06:19.512348 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511735 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511738 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511740 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511743 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511745 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511747 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511752 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511761 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511764 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511767 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511770 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511772 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511774 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511777 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511780 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511782 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511784 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511787 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511789 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511792 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 16:06:19.512801 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511794 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511797 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511799 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511801 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511804 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511807 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511809 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511811 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511814 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511817 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511821 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511823 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511826 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511828 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511831 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511836 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511839 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511842 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511845 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511847 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 16:06:19.514003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511850 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511853 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511855 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511858 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511861 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511863 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511865 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511868 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511870 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511872 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511875 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511878 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511880 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511883 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511885 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.511888 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512586 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512592 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512595 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512598 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 16:06:19.514497 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512603 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512607 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512609 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512612 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512615 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512618 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512620 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512623 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512626 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512629 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512631 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512635 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512639 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512643 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512646 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512648 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512651 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512653 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512656 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512658 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 16:06:19.514981 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512661 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512664 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512667 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512669 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512672 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512674 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512677 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512679 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512681 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512684 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512686 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512689 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512692 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512695 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512697 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512700 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512702 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512705 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512708 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512710 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 16:06:19.515508 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512713 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512715 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512718 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512720 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512723 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512725 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512728 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512731 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512733 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512736 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512738 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512740 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512743 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512745 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512748 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512750 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512753 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512756 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512758 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 16:06:19.516010 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512761 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512763 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512766 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512768 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512770 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512774 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512776 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512779 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512782 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512785 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512787 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512790 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512792 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512795 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512797 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512799 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512802 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512805 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512810 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 16:06:19.516517 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512813 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512817 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512820 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512822 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512825 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512827 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512830 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512832 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512835 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512837 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512841 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512844 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512847 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512850 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512853 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512855 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.512858 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513262 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513268 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513271 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 16:06:19.516997 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513275 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513278 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513281 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513283 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513286 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513289 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513291 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513294 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513297 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513299 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513302 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513304 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513306 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513311 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513315 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513318 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513320 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513323 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513325 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513327 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 16:06:19.517519 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513330 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513332 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513334 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513337 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513339 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513342 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513344 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513347 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513349 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513351 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513355 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513358 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513360 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513363 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513365 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513368 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513370 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513372 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513375 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513377 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 16:06:19.518003 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513380 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513382 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513385 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513388 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513390 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513393 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513396 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513398 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513401 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513403 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513406 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513408 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513410 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513413 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513415 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513418 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513420 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513423 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513427 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 16:06:19.518490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513429 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513432 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513436 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513443 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513446 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513449 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513452 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513455 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513458 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513461 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513464 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513468 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513470 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513473 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513476 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513478 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513481 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513483 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513486 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 16:06:19.518942 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513488 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513491 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513494 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513497 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513499 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513501 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513504 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513506 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513509 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513511 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513514 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513516 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513518 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513521 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513523 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513525 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513533 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.513535 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513605 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513612 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 16:06:19.519425 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513619 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513623 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513628 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513631 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513635 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513639 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513642 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513645 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513649 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513652 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513655 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513658 2560 flags.go:64] FLAG: --cgroup-root="" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513660 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513663 2560 flags.go:64] FLAG: --client-ca-file="" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513668 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513671 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513675 2560 flags.go:64] FLAG: --cluster-domain="" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513677 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513680 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513684 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513687 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513690 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513693 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 16:06:19.519908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513696 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513699 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513702 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513705 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513707 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513713 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513716 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513720 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513723 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513726 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513729 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513731 2560 flags.go:64] FLAG: --enable-server="true" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513734 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513738 2560 flags.go:64] FLAG: --event-burst="100" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513741 2560 flags.go:64] FLAG: --event-qps="50" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513744 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513747 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513750 2560 flags.go:64] FLAG: --eviction-hard="" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513753 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513756 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513759 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513762 2560 flags.go:64] FLAG: --eviction-soft="" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513764 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513767 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513770 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 16:06:19.520475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513773 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513776 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513779 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513782 2560 flags.go:64] FLAG: --feature-gates="" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513785 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513788 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513791 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513794 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513796 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513799 2560 flags.go:64] FLAG: --help="false" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513802 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-134-103.ec2.internal" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513805 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513810 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513813 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513816 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513819 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513822 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513825 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513827 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513830 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513833 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513836 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513839 2560 flags.go:64] FLAG: --kube-reserved="" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513842 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 16:06:19.521150 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513844 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513847 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513850 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513853 2560 flags.go:64] FLAG: --lock-file="" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513856 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513858 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513861 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513866 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513869 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513872 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513875 2560 flags.go:64] FLAG: --logging-format="text" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513878 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513881 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513884 2560 flags.go:64] FLAG: --manifest-url="" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513887 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513891 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513894 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513897 2560 flags.go:64] FLAG: --max-pods="110" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513900 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513903 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513910 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513914 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513916 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513919 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513922 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 16:06:19.521934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513929 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513932 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513935 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513938 2560 flags.go:64] FLAG: --pod-cidr="" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513941 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513946 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513949 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513952 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513954 2560 flags.go:64] FLAG: --port="10250" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513957 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513960 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08b1511f00ad7a7f0" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513963 2560 flags.go:64] FLAG: --qos-reserved="" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513966 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513970 2560 flags.go:64] FLAG: --register-node="true" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513972 2560 flags.go:64] FLAG: --register-with-taints="" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513976 2560 flags.go:64] FLAG: --registry-burst="10" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513979 2560 flags.go:64] FLAG: --registry-qps="5" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513982 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513984 2560 flags.go:64] FLAG: --reserved-memory="" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513988 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513991 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513994 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.513997 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514000 2560 flags.go:64] FLAG: --runonce="false" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514003 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 16:06:19.522572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514006 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514009 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514013 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514017 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514020 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514023 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514026 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514029 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514031 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514034 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514037 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514040 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514043 2560 flags.go:64] FLAG: --system-cgroups="" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514045 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514050 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514053 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514055 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514058 2560 flags.go:64] FLAG: --tls-min-version="" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514061 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514064 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514066 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514069 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514072 2560 flags.go:64] FLAG: --v="2" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514076 2560 flags.go:64] FLAG: --version="false" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514083 2560 flags.go:64] FLAG: --vmodule="" Jul 24 16:06:19.523184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514086 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.514089 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514280 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514286 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514289 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514292 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514295 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514297 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514300 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514304 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514308 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514310 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514313 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514316 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514318 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514321 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514323 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514326 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514329 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514331 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 16:06:19.523829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514334 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514336 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514339 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514341 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514344 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514346 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514349 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514352 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514354 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514357 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514360 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514362 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514365 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514368 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514371 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514373 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514376 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514378 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514381 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514383 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514386 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 16:06:19.524349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514389 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514394 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514397 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514399 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514402 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514405 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514407 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514410 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514412 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514415 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514417 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514420 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514422 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514425 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514427 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514430 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514432 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514437 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 16:06:19.524868 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514441 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514445 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514449 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514452 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514454 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514457 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514460 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514463 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514466 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514468 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514471 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514474 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514476 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514478 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514482 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514487 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514489 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514492 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514494 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514497 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 16:06:19.525425 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514499 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514502 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514506 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514509 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514512 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514515 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514518 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514521 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514523 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514526 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514528 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514531 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514533 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514536 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514538 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514541 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514544 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514546 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514548 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514551 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 16:06:19.525934 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514554 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.514556 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.515538 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.522364 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.522380 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.522398 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.522405 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522461 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522466 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522470 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522473 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522476 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522478 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522481 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522484 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 16:06:19.526470 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522487 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522489 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522492 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522494 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522497 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522499 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522502 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522504 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522507 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522509 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522512 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522514 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522517 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522519 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522522 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522524 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522527 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522529 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522532 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522535 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 16:06:19.526969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522537 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522540 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522542 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522545 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522548 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522550 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522553 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522555 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522558 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522561 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522565 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522568 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522570 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522573 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522575 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522578 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522580 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522583 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522586 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522589 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 16:06:19.527536 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522592 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522594 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522597 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522599 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522601 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522604 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522606 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522609 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522611 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522614 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522616 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522618 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522621 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522626 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522629 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522633 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522635 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522638 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522641 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 16:06:19.528338 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522644 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522647 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522649 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522652 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522655 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522658 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522660 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522662 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522665 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522667 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522670 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522672 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522674 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522677 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522679 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522682 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522684 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522686 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522689 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522691 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 16:06:19.528944 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522693 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522696 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522698 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522700 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522703 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522706 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522709 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522712 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522716 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522719 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522721 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522724 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.522729 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522837 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 16:06:19.529471 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522841 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522844 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522847 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522850 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522852 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522855 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522857 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522860 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522863 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522867 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522870 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522872 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522875 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522878 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522880 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522882 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522885 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522888 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522890 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 16:06:19.530019 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522893 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522895 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522897 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522900 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522902 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522904 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522907 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522910 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522913 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522916 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522918 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522921 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522924 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522926 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522928 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522931 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522933 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522936 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522938 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522941 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 16:06:19.530553 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522943 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522946 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522948 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522951 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522953 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522956 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522958 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522960 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522963 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522965 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522968 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522970 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522973 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522975 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522977 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522980 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522983 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522985 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522988 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522993 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 16:06:19.531048 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522995 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.522998 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523002 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523005 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523007 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523010 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523012 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523015 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523017 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523020 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523022 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523025 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523028 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523030 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523033 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523035 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523038 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523040 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523043 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523045 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 16:06:19.531577 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523048 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523050 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523052 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523055 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523057 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523060 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523062 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523065 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523067 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523070 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523072 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523077 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523080 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523083 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523088 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523092 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523095 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523098 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 16:06:19.532266 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:19.523101 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 16:06:19.532991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.523106 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 16:06:19.532991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.523208 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 16:06:19.532991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.525107 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 16:06:19.532991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.526347 2560 server.go:1019] "Starting client certificate rotation" Jul 24 16:06:19.532991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.526461 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 16:06:19.532991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.526508 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 16:06:19.532991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.528925 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 16:06:19.534506 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.534488 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 16:06:19.542355 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.542340 2560 log.go:25] "Validated CRI v1 runtime API" Jul 24 16:06:19.544091 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.544077 2560 log.go:25] "Validated CRI v1 image API" Jul 24 16:06:19.545578 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.545556 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 16:06:19.547701 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.547675 2560 fs.go:135] Filesystem UUIDs: map[3b03be57-0124-4263-b4a1-99c2fbedd033:/dev/nvme0n1p4 58e2c01b-d531-4e84-849e-0c39ea4656b7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 24 16:06:19.547760 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.547701 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 16:06:19.552514 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.552405 2560 manager.go:216] Machine: {Timestamp:2026-07-24 16:06:19.551274107 +0000 UTC m=+0.308837627 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089644 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec221a7dfaf5202246987a670a0cd24c SystemUUID:ec221a7d-faf5-2022-4698-7a670a0cd24c BootID:69c25f0a-9d10-4f90-aab8-00665a8846d4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c3:cf:97:e9:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c3:cf:97:e9:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:0e:c1:7c:61:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 16:06:19.552514 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.552510 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 16:06:19.552611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.552591 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 16:06:19.555307 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.555280 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 16:06:19.555444 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.555310 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 16:06:19.555904 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.555893 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 16:06:19.555936 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.555906 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 16:06:19.555936 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.555920 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 16:06:19.556746 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.556736 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 16:06:19.557569 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.557559 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 16:06:19.559360 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.559350 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 24 16:06:19.559465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.559456 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 16:06:19.560200 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.560190 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 24 16:06:19.560245 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.560225 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 16:06:19.560245 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.560244 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 16:06:19.560298 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.560253 2560 kubelet.go:398] "Adding apiserver pod source" Jul 24 16:06:19.560298 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.560261 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 16:06:19.560578 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.560567 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 16:06:19.560616 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.560587 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 16:06:19.562597 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.562578 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 16:06:19.564361 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.564348 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 16:06:19.564413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.564378 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 16:06:19.566668 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566656 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 16:06:19.566715 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566673 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 16:06:19.566715 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566679 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 16:06:19.566715 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566685 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 16:06:19.566715 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566693 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 16:06:19.566715 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566698 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 16:06:19.566715 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566716 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 16:06:19.566866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566722 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 16:06:19.566866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566728 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 16:06:19.566866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566733 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 16:06:19.566866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566745 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 16:06:19.566866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.566750 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 16:06:19.567382 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.567373 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 16:06:19.567382 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.567382 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 16:06:19.573171 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.573152 2560 server.go:1268] "Started kubelet" Jul 24 16:06:19.573314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.573287 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 16:06:19.573556 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.573526 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 16:06:19.573602 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.573589 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 16:06:19.577555 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.577527 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 16:06:19.578096 ip-10-0-134-103 systemd[1]: Started Kubernetes Kubelet. Jul 24 16:06:19.580430 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.578704 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 16:06:19.580430 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.578752 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 16:06:19.580430 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.579322 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-103.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 16:06:19.580430 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.579598 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 24 16:06:19.580680 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.580594 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 16:06:19.581423 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.581407 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 16:06:19.583237 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.581769 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-103.ec2.internal.18c5457124ca5cbb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-103.ec2.internal,UID:ip-10-0-134-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-103.ec2.internal,},FirstTimestamp:2026-07-24 16:06:19.573132475 +0000 UTC m=+0.330695995,LastTimestamp:2026-07-24 16:06:19.573132475 +0000 UTC m=+0.330695995,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-103.ec2.internal,}" Jul 24 16:06:19.586526 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.586512 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 16:06:19.586590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.586539 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 16:06:19.586694 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.586641 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:19.587032 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587011 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 16:06:19.587032 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587028 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 16:06:19.587170 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587063 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 16:06:19.587244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587178 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 24 16:06:19.587244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587187 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 24 16:06:19.587455 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587437 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 16:06:19.587455 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587452 2560 factory.go:55] Registering systemd factory Jul 24 16:06:19.587455 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587459 2560 factory.go:223] Registration of the systemd container factory successfully Jul 24 16:06:19.587696 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587677 2560 factory.go:153] Registering CRI-O factory Jul 24 16:06:19.587696 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587698 2560 factory.go:223] Registration of the crio container factory successfully Jul 24 16:06:19.587803 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587719 2560 factory.go:103] Registering Raw factory Jul 24 16:06:19.587803 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.587731 2560 manager.go:1196] Started watching for new ooms in manager Jul 24 16:06:19.588457 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.588444 2560 manager.go:349] Starting recovery of all containers Jul 24 16:06:19.597613 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.597508 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p55qq" Jul 24 16:06:19.598733 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.598716 2560 manager.go:354] Recovery completed Jul 24 16:06:19.602516 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.602494 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 16:06:19.603630 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.603615 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 16:06:19.604106 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.604084 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 16:06:19.607063 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.607046 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientMemory" Jul 24 16:06:19.607134 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.607082 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 16:06:19.607134 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.607098 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientPID" Jul 24 16:06:19.607655 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.607642 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 16:06:19.607655 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.607653 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 16:06:19.607785 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.607654 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 16:06:19.607785 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.607687 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 24 16:06:19.608308 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.608294 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientMemory" Jul 24 16:06:19.608375 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.608321 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 16:06:19.608375 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.608331 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientPID" Jul 24 16:06:19.609182 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.609170 2560 policy_none.go:49] "None policy: Start" Jul 24 16:06:19.609274 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.609186 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 16:06:19.609274 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.609196 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 16:06:19.610078 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.610067 2560 policy_none.go:47] "Start" Jul 24 16:06:19.610702 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.610685 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p55qq" Jul 24 16:06:19.611323 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.611254 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-103.ec2.internal.18c5457126d02026 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-103.ec2.internal,UID:ip-10-0-134-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-103.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-103.ec2.internal,},FirstTimestamp:2026-07-24 16:06:19.607064614 +0000 UTC m=+0.364628140,LastTimestamp:2026-07-24 16:06:19.607064614 +0000 UTC m=+0.364628140,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-103.ec2.internal,}" Jul 24 16:06:19.641791 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.641776 2560 manager.go:337] "Starting Device Plugin manager" Jul 24 16:06:19.641882 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.641814 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 16:06:19.641882 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.641823 2560 server.go:87] "Starting device plugin registration server" Jul 24 16:06:19.642090 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.642078 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 16:06:19.642131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.642095 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 16:06:19.642208 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.642187 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 16:06:19.642306 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.642296 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 16:06:19.642358 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.642307 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 16:06:19.643740 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.643716 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 16:06:19.643834 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.643774 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:19.742491 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.742461 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 16:06:19.743702 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.743680 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientMemory" Jul 24 16:06:19.743803 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.743719 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 16:06:19.743803 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.743733 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientPID" Jul 24 16:06:19.743803 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.743769 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-103.ec2.internal" Jul 24 16:06:19.751183 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.751167 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-103.ec2.internal" Jul 24 16:06:19.751294 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.751190 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-103.ec2.internal\": node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:19.768427 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.768365 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:19.832800 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.832765 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 16:06:19.834082 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.834060 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 16:06:19.834200 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.834098 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 16:06:19.834200 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.834120 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 16:06:19.834200 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.834151 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 16:06:19.836525 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.836505 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 16:06:19.869139 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.869106 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:19.934599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.934559 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal"] Jul 24 16:06:19.934686 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.934646 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 16:06:19.935752 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.935734 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientMemory" Jul 24 16:06:19.935842 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.935769 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 16:06:19.935842 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.935785 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientPID" Jul 24 16:06:19.937061 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.937050 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 16:06:19.937204 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.937189 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" Jul 24 16:06:19.937264 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.937236 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 16:06:19.937818 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.937799 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientMemory" Jul 24 16:06:19.937944 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.937827 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 16:06:19.937944 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.937841 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientPID" Jul 24 16:06:19.937944 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.937873 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientMemory" Jul 24 16:06:19.937944 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.937898 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 16:06:19.937944 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.937912 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientPID" Jul 24 16:06:19.939027 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.939011 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal" Jul 24 16:06:19.939116 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.939037 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 16:06:19.940169 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.940156 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientMemory" Jul 24 16:06:19.940253 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.940178 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 16:06:19.940253 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.940188 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientPID" Jul 24 16:06:19.960857 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.960840 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-103.ec2.internal\" not found" node="ip-10-0-134-103.ec2.internal" Jul 24 16:06:19.965049 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.965034 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-103.ec2.internal\" not found" node="ip-10-0-134-103.ec2.internal" Jul 24 16:06:19.969499 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:19.969485 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:19.989792 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.989762 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc717ffd274673b36048ff8694259c2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal\" (UID: \"cc717ffd274673b36048ff8694259c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" Jul 24 16:06:19.989866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.989797 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc717ffd274673b36048ff8694259c2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal\" (UID: \"cc717ffd274673b36048ff8694259c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" Jul 24 16:06:19.989866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:19.989821 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46b3df3b80b90b643851b1c22d7ab10a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-103.ec2.internal\" (UID: \"46b3df3b80b90b643851b1c22d7ab10a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal" Jul 24 16:06:20.069951 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:20.069930 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:20.090330 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:20.090309 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc717ffd274673b36048ff8694259c2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal\" (UID: \"cc717ffd274673b36048ff8694259c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" Jul 24 16:06:20.090398 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:20.090325 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc717ffd274673b36048ff8694259c2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal\" (UID: \"cc717ffd274673b36048ff8694259c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" Jul 24 16:06:20.090398 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:20.090346 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc717ffd274673b36048ff8694259c2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal\" (UID: \"cc717ffd274673b36048ff8694259c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" Jul 24 16:06:20.090398 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:20.090363 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46b3df3b80b90b643851b1c22d7ab10a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-103.ec2.internal\" (UID: \"46b3df3b80b90b643851b1c22d7ab10a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal" Jul 24 16:06:20.090500 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:20.090395 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46b3df3b80b90b643851b1c22d7ab10a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-103.ec2.internal\" (UID: \"46b3df3b80b90b643851b1c22d7ab10a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal" Jul 24 16:06:20.090500 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:20.090449 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc717ffd274673b36048ff8694259c2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal\" (UID: \"cc717ffd274673b36048ff8694259c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" Jul 24 16:06:20.170465 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:20.170431 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:20.262900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:20.262877 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" Jul 24 16:06:20.268542 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:20.268524 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal" Jul 24 16:06:20.270662 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:20.270646 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:20.371383 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:20.371324 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:20.471838 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:20.471802 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:21.852270 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:21.852250 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 16:06:21.852662 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:21.852397 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 16:06:21.898699 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:21.898664 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:21.938407 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:21.938385 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 16:06:21.941476 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:21.941448 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 16:01:20 +0000 UTC" deadline="2028-03-30 16:25:52.484895949 +0000 UTC" Jul 24 16:06:21.941540 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:21.941476 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14760h19m30.54342271s" Jul 24 16:06:21.965557 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:21.965505 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 16:06:21.999499 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:21.999478 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:22.050913 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.050888 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 16:06:22.054592 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.054579 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientMemory" Jul 24 16:06:22.054664 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.054608 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 16:06:22.054664 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.054618 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeHasSufficientPID" Jul 24 16:06:22.100464 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:22.100442 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:22.201289 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:22.201258 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:22.213889 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:22.213858 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b3df3b80b90b643851b1c22d7ab10a.slice/crio-4de0118bef36178104ded67b44fc3eb74e26f9a1712ce145b72b3b20a7c8fa06 WatchSource:0}: Error finding container 4de0118bef36178104ded67b44fc3eb74e26f9a1712ce145b72b3b20a7c8fa06: Status 404 returned error can't find the container with id 4de0118bef36178104ded67b44fc3eb74e26f9a1712ce145b72b3b20a7c8fa06 Jul 24 16:06:22.214490 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:22.214443 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc717ffd274673b36048ff8694259c2f.slice/crio-caee6dafd0ca417f275f057550cb63776c76a43bfda49c06fc74970938178da6 WatchSource:0}: Error finding container caee6dafd0ca417f275f057550cb63776c76a43bfda49c06fc74970938178da6: Status 404 returned error can't find the container with id caee6dafd0ca417f275f057550cb63776c76a43bfda49c06fc74970938178da6 Jul 24 16:06:22.220266 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.220253 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:06:22.302297 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:22.302262 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-103.ec2.internal\" not found" Jul 24 16:06:22.326948 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.326930 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 16:06:22.413638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.413615 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" Jul 24 16:06:22.424901 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.424884 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 16:06:22.425689 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.425677 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal" Jul 24 16:06:22.434085 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.434071 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 16:06:22.887908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.887878 2560 apiserver.go:52] "Watching apiserver" Jul 24 16:06:22.894089 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.894064 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 16:06:22.896277 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.896249 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4zmmf","openshift-dns/node-resolver-xwds8","openshift-multus/multus-mlv5n","openshift-multus/network-metrics-daemon-9lbnz","openshift-network-diagnostics/network-check-target-42m7w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8","openshift-image-registry/node-ca-hlv94","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal","openshift-multus/multus-additional-cni-plugins-kf2n5","openshift-network-operator/iptables-alerter-67vvt","openshift-ovn-kubernetes/ovnkube-node-tlf24","kube-system/global-pull-secret-syncer-7gtd8","kube-system/konnectivity-agent-87tpp"] Jul 24 16:06:22.898241 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.897903 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:22.899157 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.899117 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:22.899253 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:22.899212 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:22.900900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.900206 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:22.900900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.900363 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 16:06:22.900900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.900561 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 16:06:22.901087 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.900910 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 16:06:22.901087 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.900908 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbfzs\"" Jul 24 16:06:22.904362 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.904344 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 16:06:22.904680 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.904663 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bc7wh\"" Jul 24 16:06:22.904860 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.904846 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 16:06:22.906169 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.905457 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:22.906169 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:22.905516 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:22.906169 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.905723 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.906991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.906429 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:22.906991 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:22.906499 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:22.906991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.906511 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:22.906991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.906526 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:22.907341 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.907326 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.908020 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.907997 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 16:06:22.908300 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.908281 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zgc57\"" Jul 24 16:06:22.908976 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.908956 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:22.909110 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.909010 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 16:06:22.909191 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.908964 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 16:06:22.909311 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.909287 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 16:06:22.909542 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.909517 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 16:06:22.909722 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.909706 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 16:06:22.909872 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.909830 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6pqng\"" Jul 24 16:06:22.910137 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.910115 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 16:06:22.912348 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.912326 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 16:06:22.912579 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.912563 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9k9zm\"" Jul 24 16:06:22.912799 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.912785 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 16:06:22.912996 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.912977 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 16:06:22.915169 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.915081 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cvrq5\"" Jul 24 16:06:22.915347 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.915327 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 16:06:22.916237 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.916218 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 16:06:22.917669 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.917651 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:22.918040 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.918018 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 16:06:22.918119 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.918094 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.918119 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.918098 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 16:06:22.918630 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.918612 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mhk85\"" Jul 24 16:06:22.920836 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.920813 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 16:06:22.921104 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.921085 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 16:06:22.921254 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.921224 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 16:06:22.921334 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.921307 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 16:06:22.921518 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.921501 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 16:06:22.921965 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.921928 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 16:06:22.922092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.922075 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 16:06:22.922399 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.922379 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wph7q\"" Jul 24 16:06:22.922472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.922413 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bl26r\"" Jul 24 16:06:22.922578 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.922561 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 16:06:22.923272 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.923256 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 16:06:22.923866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.923835 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-system-cni-dir\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:22.923951 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.923889 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7c7\" (UniqueName: \"kubernetes.io/projected/95229c4e-2eb9-4401-8282-0cfd32c54496-kube-api-access-kw7c7\") pod \"node-ca-hlv94\" (UID: \"95229c4e-2eb9-4401-8282-0cfd32c54496\") " pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:22.923951 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.923916 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-var-lib-openvswitch\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.924052 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.923949 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-run-netns\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.924052 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.923990 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f746255-6327-410b-82e0-2a3b60e801fb-konnectivity-ca\") pod \"konnectivity-agent-87tpp\" (UID: \"6f746255-6327-410b-82e0-2a3b60e801fb\") " pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:22.924052 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924010 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-systemd-units\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.924052 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924025 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9ef7edf-3ea8-4270-b7c7-acf95e550718-ovn-node-metrics-cert\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.924052 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924047 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-socket-dir-parent\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924071 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924094 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-registration-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924118 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924163 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-run-k8s-cni-cncf-io\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924187 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qll2\" (UniqueName: \"kubernetes.io/projected/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-kube-api-access-9qll2\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924209 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-kubernetes\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924243 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-slash\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924267 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9ef7edf-3ea8-4270-b7c7-acf95e550718-ovnkube-script-lib\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924289 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65297386-a516-4da5-b689-c3e03c004310-hosts-file\") pod \"node-resolver-xwds8\" (UID: \"65297386-a516-4da5-b689-c3e03c004310\") " pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:22.924314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924311 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-sysctl-d\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924347 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-tmp\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924380 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9ef7edf-3ea8-4270-b7c7-acf95e550718-env-overrides\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924402 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-hostroot\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924431 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924465 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-cnibin\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924487 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-socket-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924531 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-sys-fs\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924581 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65297386-a516-4da5-b689-c3e03c004310-tmp-dir\") pod \"node-resolver-xwds8\" (UID: \"65297386-a516-4da5-b689-c3e03c004310\") " pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924614 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6tv\" (UniqueName: \"kubernetes.io/projected/cd77de22-56d1-4095-8674-7613919037b0-kube-api-access-fz6tv\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924654 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f746255-6327-410b-82e0-2a3b60e801fb-agent-certs\") pod \"konnectivity-agent-87tpp\" (UID: \"6f746255-6327-410b-82e0-2a3b60e801fb\") " pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924677 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-sysctl-conf\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924701 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-run-ovn\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924735 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.924778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924768 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-var-lib-cni-multus\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924804 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-systemd\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924834 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-log-socket\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924869 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9ef7edf-3ea8-4270-b7c7-acf95e550718-ovnkube-config\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924896 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-system-cni-dir\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924911 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-cni-binary-copy\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924935 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924959 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.924981 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-kubelet\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925012 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-run-netns\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925041 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-cni-netd\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925064 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-kubelet-config\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925090 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-dbus\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925127 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-cnibin\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925176 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8a4620c-a9c2-49f8-baa6-3806c4c9434c-iptables-alerter-script\") pod \"iptables-alerter-67vvt\" (UID: \"a8a4620c-a9c2-49f8-baa6-3806c4c9434c\") " pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925198 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8a4620c-a9c2-49f8-baa6-3806c4c9434c-host-slash\") pod \"iptables-alerter-67vvt\" (UID: \"a8a4620c-a9c2-49f8-baa6-3806c4c9434c\") " pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:22.925432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925213 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-lib-modules\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925240 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-run-systemd\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925266 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925290 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhhd2\" (UniqueName: \"kubernetes.io/projected/b9ef7edf-3ea8-4270-b7c7-acf95e550718-kube-api-access-lhhd2\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925311 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-var-lib-cni-bin\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925339 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlshr\" (UniqueName: \"kubernetes.io/projected/a8a4620c-a9c2-49f8-baa6-3806c4c9434c-kube-api-access-mlshr\") pod \"iptables-alerter-67vvt\" (UID: \"a8a4620c-a9c2-49f8-baa6-3806c4c9434c\") " pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925359 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72rn\" (UniqueName: \"kubernetes.io/projected/665dc859-e7eb-4fa5-8756-ced75a40e249-kube-api-access-s72rn\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925371 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-sys\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925386 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-run-openvswitch\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925408 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-daemon-config\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925462 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-os-release\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925489 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925554 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925576 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95229c4e-2eb9-4401-8282-0cfd32c54496-serviceca\") pod \"node-ca-hlv94\" (UID: \"95229c4e-2eb9-4401-8282-0cfd32c54496\") " pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925599 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-run\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925620 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-var-lib-kubelet\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.926172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925652 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-os-release\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925676 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-etc-selinux\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925701 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52q5n\" (UniqueName: \"kubernetes.io/projected/65297386-a516-4da5-b689-c3e03c004310-kube-api-access-52q5n\") pod \"node-resolver-xwds8\" (UID: \"65297386-a516-4da5-b689-c3e03c004310\") " pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95229c4e-2eb9-4401-8282-0cfd32c54496-host\") pod \"node-ca-hlv94\" (UID: \"95229c4e-2eb9-4401-8282-0cfd32c54496\") " pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925758 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-sysconfig\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925802 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-cni-dir\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925844 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-var-lib-kubelet\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-conf-dir\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-etc-kubernetes\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925912 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmfk7\" (UniqueName: \"kubernetes.io/projected/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-kube-api-access-dmfk7\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925959 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-node-log\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.925988 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5pw9\" (UniqueName: \"kubernetes.io/projected/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-kube-api-access-z5pw9\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.926009 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-device-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.926048 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-modprobe-d\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.926362 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-host\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.926384 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-tuned\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:22.926975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.926408 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-etc-openvswitch\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.927796 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.926430 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-cni-bin\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:22.927796 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.926453 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-run-multus-certs\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:22.927796 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.926774 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 16:06:22.951637 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.951618 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-98x5q" Jul 24 16:06:22.957430 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:22.957407 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-98x5q" Jul 24 16:06:23.014492 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.014461 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 16:06:23.027082 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027049 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-log-socket\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.027082 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027076 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9ef7edf-3ea8-4270-b7c7-acf95e550718-ovnkube-config\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.027296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027093 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-system-cni-dir\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.027296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027107 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-cni-binary-copy\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.027296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027120 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.027296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027161 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:23.027296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027187 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-kubelet\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.027296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027210 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-run-netns\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.027296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027234 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-cni-netd\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.027296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027258 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-kubelet-config\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:23.027296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027281 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-dbus\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027302 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-cnibin\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027341 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8a4620c-a9c2-49f8-baa6-3806c4c9434c-iptables-alerter-script\") pod \"iptables-alerter-67vvt\" (UID: \"a8a4620c-a9c2-49f8-baa6-3806c4c9434c\") " pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027387 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8a4620c-a9c2-49f8-baa6-3806c4c9434c-host-slash\") pod \"iptables-alerter-67vvt\" (UID: \"a8a4620c-a9c2-49f8-baa6-3806c4c9434c\") " pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027410 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-lib-modules\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027432 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-run-systemd\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027455 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027482 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhhd2\" (UniqueName: \"kubernetes.io/projected/b9ef7edf-3ea8-4270-b7c7-acf95e550718-kube-api-access-lhhd2\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027506 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-var-lib-cni-bin\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027532 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlshr\" (UniqueName: \"kubernetes.io/projected/a8a4620c-a9c2-49f8-baa6-3806c4c9434c-kube-api-access-mlshr\") pod \"iptables-alerter-67vvt\" (UID: \"a8a4620c-a9c2-49f8-baa6-3806c4c9434c\") " pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027550 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72rn\" (UniqueName: \"kubernetes.io/projected/665dc859-e7eb-4fa5-8756-ced75a40e249-kube-api-access-s72rn\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027563 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-sys\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027576 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-run-openvswitch\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-daemon-config\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.027619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027622 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-os-release\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027638 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027657 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027678 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95229c4e-2eb9-4401-8282-0cfd32c54496-serviceca\") pod \"node-ca-hlv94\" (UID: \"95229c4e-2eb9-4401-8282-0cfd32c54496\") " pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-run\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027722 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-var-lib-kubelet\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027745 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-os-release\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027769 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-etc-selinux\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027793 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52q5n\" (UniqueName: \"kubernetes.io/projected/65297386-a516-4da5-b689-c3e03c004310-kube-api-access-52q5n\") pod \"node-resolver-xwds8\" (UID: \"65297386-a516-4da5-b689-c3e03c004310\") " pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027818 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95229c4e-2eb9-4401-8282-0cfd32c54496-host\") pod \"node-ca-hlv94\" (UID: \"95229c4e-2eb9-4401-8282-0cfd32c54496\") " pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027843 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-sysconfig\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027867 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-cni-dir\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027897 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-var-lib-kubelet\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-conf-dir\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027955 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-etc-kubernetes\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.027981 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028006 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmfk7\" (UniqueName: \"kubernetes.io/projected/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-kube-api-access-dmfk7\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.028179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028030 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-node-log\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028054 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5pw9\" (UniqueName: \"kubernetes.io/projected/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-kube-api-access-z5pw9\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028079 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-device-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028102 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-modprobe-d\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028127 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-host\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028175 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-tuned\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028201 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-etc-openvswitch\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-cni-bin\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028254 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-run-multus-certs\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028279 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-system-cni-dir\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028304 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7c7\" (UniqueName: \"kubernetes.io/projected/95229c4e-2eb9-4401-8282-0cfd32c54496-kube-api-access-kw7c7\") pod \"node-ca-hlv94\" (UID: \"95229c4e-2eb9-4401-8282-0cfd32c54496\") " pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028327 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-var-lib-openvswitch\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028356 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-run-netns\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028384 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f746255-6327-410b-82e0-2a3b60e801fb-konnectivity-ca\") pod \"konnectivity-agent-87tpp\" (UID: \"6f746255-6327-410b-82e0-2a3b60e801fb\") " pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028412 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-systemd-units\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9ef7edf-3ea8-4270-b7c7-acf95e550718-ovn-node-metrics-cert\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028462 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-socket-dir-parent\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.028945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028489 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028514 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-registration-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028539 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028565 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-run-k8s-cni-cncf-io\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028591 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qll2\" (UniqueName: \"kubernetes.io/projected/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-kube-api-access-9qll2\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028616 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-kubernetes\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028648 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-slash\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028674 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9ef7edf-3ea8-4270-b7c7-acf95e550718-ovnkube-script-lib\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65297386-a516-4da5-b689-c3e03c004310-hosts-file\") pod \"node-resolver-xwds8\" (UID: \"65297386-a516-4da5-b689-c3e03c004310\") " pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028723 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-sysctl-d\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028760 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-tmp\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028785 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9ef7edf-3ea8-4270-b7c7-acf95e550718-env-overrides\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028809 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-hostroot\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028838 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028869 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-cnibin\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028897 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-socket-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028921 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-sys-fs\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.032434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028953 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65297386-a516-4da5-b689-c3e03c004310-tmp-dir\") pod \"node-resolver-xwds8\" (UID: \"65297386-a516-4da5-b689-c3e03c004310\") " pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.028982 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6tv\" (UniqueName: \"kubernetes.io/projected/cd77de22-56d1-4095-8674-7613919037b0-kube-api-access-fz6tv\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029004 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-log-socket\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029009 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f746255-6327-410b-82e0-2a3b60e801fb-agent-certs\") pod \"konnectivity-agent-87tpp\" (UID: \"6f746255-6327-410b-82e0-2a3b60e801fb\") " pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029059 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-sysctl-conf\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029125 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-run-ovn\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029174 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029204 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-var-lib-cni-multus\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029235 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-systemd\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029286 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-systemd\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029811 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029891 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9ef7edf-3ea8-4270-b7c7-acf95e550718-ovnkube-config\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.029987 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-system-cni-dir\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030194 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-kubelet-config\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030489 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-cni-binary-copy\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030595 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-dbus\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030623 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f746255-6327-410b-82e0-2a3b60e801fb-konnectivity-ca\") pod \"konnectivity-agent-87tpp\" (UID: \"6f746255-6327-410b-82e0-2a3b60e801fb\") " pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:23.033267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030640 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-cnibin\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030688 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95229c4e-2eb9-4401-8282-0cfd32c54496-host\") pod \"node-ca-hlv94\" (UID: \"95229c4e-2eb9-4401-8282-0cfd32c54496\") " pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030714 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030728 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8a4620c-a9c2-49f8-baa6-3806c4c9434c-host-slash\") pod \"iptables-alerter-67vvt\" (UID: \"a8a4620c-a9c2-49f8-baa6-3806c4c9434c\") " pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030758 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-lib-modules\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030779 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-sysconfig\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030812 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-run-systemd\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030831 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-host\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030862 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030873 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-modprobe-d\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030922 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-var-lib-cni-bin\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030927 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-var-lib-kubelet\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.030994 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-cni-dir\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031173 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-sys\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031221 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-run-openvswitch\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031254 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-etc-kubernetes\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031305 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9ef7edf-3ea8-4270-b7c7-acf95e550718-ovnkube-script-lib\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031372 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-systemd-units\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.034111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031671 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031688 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-daemon-config\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031713 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95229c4e-2eb9-4401-8282-0cfd32c54496-serviceca\") pod \"node-ca-hlv94\" (UID: \"95229c4e-2eb9-4401-8282-0cfd32c54496\") " pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031729 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-var-lib-kubelet\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031784 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-run\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031822 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031845 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-os-release\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031847 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-sysctl-conf\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031896 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031902 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-node-log\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031916 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-registration-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031910 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-run-k8s-cni-cncf-io\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031672 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-etc-selinux\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031959 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-system-cni-dir\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031963 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-var-lib-cni-multus\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031988 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-etc-openvswitch\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031990 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-device-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.031993 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-run-netns\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.034966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032008 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-run-ovn\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032025 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-kubelet\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032033 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-conf-dir\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032060 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-multus-socket-dir-parent\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032067 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-var-lib-openvswitch\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032122 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-cni-bin\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032182 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-run-multus-certs\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.032236 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.032281 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret podName:6ea154fa-ed3f-4f3c-b3d4-980666719a5d nodeName:}" failed. No retries permitted until 2026-07-24 16:06:23.532266645 +0000 UTC m=+2.963901984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret") pod "global-pull-secret-syncer-7gtd8" (UID: "6ea154fa-ed3f-4f3c-b3d4-980666719a5d") : object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032294 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-sys-fs\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032340 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-cnibin\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032371 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032379 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65297386-a516-4da5-b689-c3e03c004310-hosts-file\") pod \"node-resolver-xwds8\" (UID: \"65297386-a516-4da5-b689-c3e03c004310\") " pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-os-release\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032411 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-cni-netd\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032441 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-host-run-netns\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032439 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/665dc859-e7eb-4fa5-8756-ced75a40e249-socket-dir\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.035840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032483 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-sysctl-d\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.036809 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032483 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-kubernetes\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.036809 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032608 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65297386-a516-4da5-b689-c3e03c004310-tmp-dir\") pod \"node-resolver-xwds8\" (UID: \"65297386-a516-4da5-b689-c3e03c004310\") " pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:23.036809 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032695 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-hostroot\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.036809 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.032733 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:23.036809 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.032743 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9ef7edf-3ea8-4270-b7c7-acf95e550718-host-slash\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.037501 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.037480 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs podName:cd77de22-56d1-4095-8674-7613919037b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:23.537452167 +0000 UTC m=+2.969087523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs") pod "network-metrics-daemon-9lbnz" (UID: "cd77de22-56d1-4095-8674-7613919037b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:23.050126 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.050084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9ef7edf-3ea8-4270-b7c7-acf95e550718-ovn-node-metrics-cert\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.050353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.050333 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-etc-tuned\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.050613 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.050590 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9ef7edf-3ea8-4270-b7c7-acf95e550718-env-overrides\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.051809 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.051778 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f746255-6327-410b-82e0-2a3b60e801fb-agent-certs\") pod \"konnectivity-agent-87tpp\" (UID: \"6f746255-6327-410b-82e0-2a3b60e801fb\") " pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:23.052076 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.052053 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8a4620c-a9c2-49f8-baa6-3806c4c9434c-iptables-alerter-script\") pod \"iptables-alerter-67vvt\" (UID: \"a8a4620c-a9c2-49f8-baa6-3806c4c9434c\") " pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:23.053558 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.053536 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-tmp\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.054347 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.054119 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 16:06:23.054347 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.054157 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 16:06:23.054347 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.054172 2560 projected.go:196] Error preparing data for projected volume kube-api-access-mhcx6 for pod openshift-network-diagnostics/network-check-target-42m7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:23.054347 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.054226 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6 podName:02425e0f-39d9-47b3-9a4c-c0504676aad1 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:23.55421003 +0000 UTC m=+2.985845370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mhcx6" (UniqueName: "kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6") pod "network-check-target-42m7w" (UID: "02425e0f-39d9-47b3-9a4c-c0504676aad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:23.057700 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.057639 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlshr\" (UniqueName: \"kubernetes.io/projected/a8a4620c-a9c2-49f8-baa6-3806c4c9434c-kube-api-access-mlshr\") pod \"iptables-alerter-67vvt\" (UID: \"a8a4620c-a9c2-49f8-baa6-3806c4c9434c\") " pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:23.058254 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.058231 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52q5n\" (UniqueName: \"kubernetes.io/projected/65297386-a516-4da5-b689-c3e03c004310-kube-api-access-52q5n\") pod \"node-resolver-xwds8\" (UID: \"65297386-a516-4da5-b689-c3e03c004310\") " pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:23.058979 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.058953 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5pw9\" (UniqueName: \"kubernetes.io/projected/4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d-kube-api-access-z5pw9\") pod \"multus-mlv5n\" (UID: \"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d\") " pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.060235 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.060202 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7c7\" (UniqueName: \"kubernetes.io/projected/95229c4e-2eb9-4401-8282-0cfd32c54496-kube-api-access-kw7c7\") pod \"node-ca-hlv94\" (UID: \"95229c4e-2eb9-4401-8282-0cfd32c54496\") " pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:23.060395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.060374 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6tv\" (UniqueName: \"kubernetes.io/projected/cd77de22-56d1-4095-8674-7613919037b0-kube-api-access-fz6tv\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:23.060779 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.060760 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmfk7\" (UniqueName: \"kubernetes.io/projected/a6006b0f-b1d2-47b8-9d92-0c4d60d91d88-kube-api-access-dmfk7\") pod \"tuned-4zmmf\" (UID: \"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88\") " pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.060839 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.060813 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhhd2\" (UniqueName: \"kubernetes.io/projected/b9ef7edf-3ea8-4270-b7c7-acf95e550718-kube-api-access-lhhd2\") pod \"ovnkube-node-tlf24\" (UID: \"b9ef7edf-3ea8-4270-b7c7-acf95e550718\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.061783 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.061757 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72rn\" (UniqueName: \"kubernetes.io/projected/665dc859-e7eb-4fa5-8756-ced75a40e249-kube-api-access-s72rn\") pod \"aws-ebs-csi-driver-node-hl5k8\" (UID: \"665dc859-e7eb-4fa5-8756-ced75a40e249\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.062648 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.062626 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qll2\" (UniqueName: \"kubernetes.io/projected/c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5-kube-api-access-9qll2\") pod \"multus-additional-cni-plugins-kf2n5\" (UID: \"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5\") " pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.165732 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.165639 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" event={"ID":"cc717ffd274673b36048ff8694259c2f","Type":"ContainerStarted","Data":"caee6dafd0ca417f275f057550cb63776c76a43bfda49c06fc74970938178da6"} Jul 24 16:06:23.167314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.167280 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal" event={"ID":"46b3df3b80b90b643851b1c22d7ab10a","Type":"ContainerStarted","Data":"4de0118bef36178104ded67b44fc3eb74e26f9a1712ce145b72b3b20a7c8fa06"} Jul 24 16:06:23.218934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.218902 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 16:06:23.223025 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.222999 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" Jul 24 16:06:23.232205 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.232016 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xwds8" Jul 24 16:06:23.232824 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:23.232797 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665dc859_e7eb_4fa5_8756_ced75a40e249.slice/crio-3762979905df655640514b4033a1208836b9157d4376912e7e030bae1ea9a268 WatchSource:0}: Error finding container 3762979905df655640514b4033a1208836b9157d4376912e7e030bae1ea9a268: Status 404 returned error can't find the container with id 3762979905df655640514b4033a1208836b9157d4376912e7e030bae1ea9a268 Jul 24 16:06:23.240139 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.240117 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mlv5n" Jul 24 16:06:23.241915 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:23.241886 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65297386_a516_4da5_b689_c3e03c004310.slice/crio-bd357c44f96efc5482a58e275acb546219a43e17480adb4cbcdffeb9234cbc35 WatchSource:0}: Error finding container bd357c44f96efc5482a58e275acb546219a43e17480adb4cbcdffeb9234cbc35: Status 404 returned error can't find the container with id bd357c44f96efc5482a58e275acb546219a43e17480adb4cbcdffeb9234cbc35 Jul 24 16:06:23.248053 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.248020 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlv94" Jul 24 16:06:23.248729 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:23.248697 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffb9abd_d6d8_43d2_a3ef_23bd8d1ac46d.slice/crio-6a5b23b1b1be4f102b0cbb12aedf78d43728a68c5add53b2750be9e388916257 WatchSource:0}: Error finding container 6a5b23b1b1be4f102b0cbb12aedf78d43728a68c5add53b2750be9e388916257: Status 404 returned error can't find the container with id 6a5b23b1b1be4f102b0cbb12aedf78d43728a68c5add53b2750be9e388916257 Jul 24 16:06:23.255030 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.255011 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:23.258132 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:23.258107 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95229c4e_2eb9_4401_8282_0cfd32c54496.slice/crio-295eb46e847c6b5e33b364bdaeacbf15bead851da0fedc8f72251ab036a0391a WatchSource:0}: Error finding container 295eb46e847c6b5e33b364bdaeacbf15bead851da0fedc8f72251ab036a0391a: Status 404 returned error can't find the container with id 295eb46e847c6b5e33b364bdaeacbf15bead851da0fedc8f72251ab036a0391a Jul 24 16:06:23.262125 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.262107 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" Jul 24 16:06:23.271456 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.271439 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" Jul 24 16:06:23.279999 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.279781 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-67vvt" Jul 24 16:06:23.288300 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.288279 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:23.533321 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.533246 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:23.533462 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.533413 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:23.533524 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.533483 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret podName:6ea154fa-ed3f-4f3c-b3d4-980666719a5d nodeName:}" failed. No retries permitted until 2026-07-24 16:06:24.533468022 +0000 UTC m=+3.965103369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret") pod "global-pull-secret-syncer-7gtd8" (UID: "6ea154fa-ed3f-4f3c-b3d4-980666719a5d") : object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:23.634384 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.634356 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:23.634557 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.634394 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:23.634557 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.634500 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:23.634557 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.634528 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 16:06:23.634557 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.634557 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 16:06:23.634737 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.634565 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs podName:cd77de22-56d1-4095-8674-7613919037b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:24.634543727 +0000 UTC m=+4.066179070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs") pod "network-metrics-daemon-9lbnz" (UID: "cd77de22-56d1-4095-8674-7613919037b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:23.634737 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.634570 2560 projected.go:196] Error preparing data for projected volume kube-api-access-mhcx6 for pod openshift-network-diagnostics/network-check-target-42m7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:23.634737 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:23.634623 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6 podName:02425e0f-39d9-47b3-9a4c-c0504676aad1 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:24.634607354 +0000 UTC m=+4.066242693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhcx6" (UniqueName: "kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6") pod "network-check-target-42m7w" (UID: "02425e0f-39d9-47b3-9a4c-c0504676aad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:23.959806 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.959727 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 16:01:22 +0000 UTC" deadline="2028-01-24 18:10:27.099289382 +0000 UTC" Jul 24 16:06:23.959806 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:23.959758 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13178h4m3.139534639s" Jul 24 16:06:24.161077 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.161039 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:24.161266 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:24.161184 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:24.170409 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.170381 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlv94" event={"ID":"95229c4e-2eb9-4401-8282-0cfd32c54496","Type":"ContainerStarted","Data":"295eb46e847c6b5e33b364bdaeacbf15bead851da0fedc8f72251ab036a0391a"} Jul 24 16:06:24.171819 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.171793 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mlv5n" event={"ID":"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d","Type":"ContainerStarted","Data":"6a5b23b1b1be4f102b0cbb12aedf78d43728a68c5add53b2750be9e388916257"} Jul 24 16:06:24.173062 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.173037 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xwds8" event={"ID":"65297386-a516-4da5-b689-c3e03c004310","Type":"ContainerStarted","Data":"bd357c44f96efc5482a58e275acb546219a43e17480adb4cbcdffeb9234cbc35"} Jul 24 16:06:24.174319 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.174296 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" event={"ID":"665dc859-e7eb-4fa5-8756-ced75a40e249","Type":"ContainerStarted","Data":"3762979905df655640514b4033a1208836b9157d4376912e7e030bae1ea9a268"} Jul 24 16:06:24.274364 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:24.274334 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3dd5400_8bd9_4cda_ab6d_fbaed2420bf5.slice/crio-96a9245b8a72950fd5681f9950e8e73e3a497b10a44883f780dfadd48f7f778d WatchSource:0}: Error finding container 96a9245b8a72950fd5681f9950e8e73e3a497b10a44883f780dfadd48f7f778d: Status 404 returned error can't find the container with id 96a9245b8a72950fd5681f9950e8e73e3a497b10a44883f780dfadd48f7f778d Jul 24 16:06:24.274875 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:24.274761 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f746255_6327_410b_82e0_2a3b60e801fb.slice/crio-8528a4a72ccaf157dbb5c07a706bf62dbb0cb17e3ad8057bb1ff9c0444010e9a WatchSource:0}: Error finding container 8528a4a72ccaf157dbb5c07a706bf62dbb0cb17e3ad8057bb1ff9c0444010e9a: Status 404 returned error can't find the container with id 8528a4a72ccaf157dbb5c07a706bf62dbb0cb17e3ad8057bb1ff9c0444010e9a Jul 24 16:06:24.276438 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:24.276363 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ef7edf_3ea8_4270_b7c7_acf95e550718.slice/crio-06bdf537eca5dfa6335dd041093a9bb85fcdd6af122c4e37495d9c7613338b7f WatchSource:0}: Error finding container 06bdf537eca5dfa6335dd041093a9bb85fcdd6af122c4e37495d9c7613338b7f: Status 404 returned error can't find the container with id 06bdf537eca5dfa6335dd041093a9bb85fcdd6af122c4e37495d9c7613338b7f Jul 24 16:06:24.277207 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:06:24.277174 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a4620c_a9c2_49f8_baa6_3806c4c9434c.slice/crio-0718cedef56b008f3b99f2f804d17ff359036c66a980dd82e137e8de74a66358 WatchSource:0}: Error finding container 0718cedef56b008f3b99f2f804d17ff359036c66a980dd82e137e8de74a66358: Status 404 returned error can't find the container with id 0718cedef56b008f3b99f2f804d17ff359036c66a980dd82e137e8de74a66358 Jul 24 16:06:24.541586 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.541411 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:24.541740 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:24.541706 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:24.541809 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:24.541754 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret podName:6ea154fa-ed3f-4f3c-b3d4-980666719a5d nodeName:}" failed. No retries permitted until 2026-07-24 16:06:26.541742195 +0000 UTC m=+5.973377534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret") pod "global-pull-secret-syncer-7gtd8" (UID: "6ea154fa-ed3f-4f3c-b3d4-980666719a5d") : object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:24.642272 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.642239 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:24.642401 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.642298 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:24.642470 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:24.642423 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:24.642518 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:24.642483 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs podName:cd77de22-56d1-4095-8674-7613919037b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:26.642465263 +0000 UTC m=+6.074100603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs") pod "network-metrics-daemon-9lbnz" (UID: "cd77de22-56d1-4095-8674-7613919037b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:24.642713 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:24.642689 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 16:06:24.642774 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:24.642717 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 16:06:24.642774 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:24.642730 2560 projected.go:196] Error preparing data for projected volume kube-api-access-mhcx6 for pod openshift-network-diagnostics/network-check-target-42m7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:24.642869 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:24.642778 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6 podName:02425e0f-39d9-47b3-9a4c-c0504676aad1 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:26.642762661 +0000 UTC m=+6.074398014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhcx6" (UniqueName: "kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6") pod "network-check-target-42m7w" (UID: "02425e0f-39d9-47b3-9a4c-c0504676aad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:24.960187 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.960098 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 16:01:22 +0000 UTC" deadline="2028-01-22 14:21:22.908909502 +0000 UTC" Jul 24 16:06:24.960187 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:24.960170 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13126h14m57.948743512s" Jul 24 16:06:25.161310 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.161283 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:25.161488 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:25.161422 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:25.163329 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.163304 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:25.163422 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:25.163407 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:25.190051 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.190008 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal" event={"ID":"46b3df3b80b90b643851b1c22d7ab10a","Type":"ContainerStarted","Data":"8e1586d83e8be46dd50f54cb63ab43693f3d898263aae7e5a7cb18f3314056c1"} Jul 24 16:06:25.192918 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.192891 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-67vvt" event={"ID":"a8a4620c-a9c2-49f8-baa6-3806c4c9434c","Type":"ContainerStarted","Data":"0718cedef56b008f3b99f2f804d17ff359036c66a980dd82e137e8de74a66358"} Jul 24 16:06:25.196693 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.196665 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerStarted","Data":"06bdf537eca5dfa6335dd041093a9bb85fcdd6af122c4e37495d9c7613338b7f"} Jul 24 16:06:25.200199 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.200114 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" event={"ID":"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88","Type":"ContainerStarted","Data":"f7c78ea04f058234200b459be2e88b98fa41bed34989f6bd8ec9d41f6071b7f3"} Jul 24 16:06:25.219800 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.218939 2560 generic.go:356] "Generic (PLEG): container finished" podID="cc717ffd274673b36048ff8694259c2f" containerID="ab0fd63650137f5c54d80b0bb7b546a18623a4e2ebf031e84f385f998aaa0b30" exitCode=0 Jul 24 16:06:25.219800 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.219024 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" event={"ID":"cc717ffd274673b36048ff8694259c2f","Type":"ContainerDied","Data":"ab0fd63650137f5c54d80b0bb7b546a18623a4e2ebf031e84f385f998aaa0b30"} Jul 24 16:06:25.228656 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.228627 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" event={"ID":"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5","Type":"ContainerStarted","Data":"96a9245b8a72950fd5681f9950e8e73e3a497b10a44883f780dfadd48f7f778d"} Jul 24 16:06:25.230893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.230870 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-87tpp" event={"ID":"6f746255-6327-410b-82e0-2a3b60e801fb","Type":"ContainerStarted","Data":"8528a4a72ccaf157dbb5c07a706bf62dbb0cb17e3ad8057bb1ff9c0444010e9a"} Jul 24 16:06:25.235658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:25.235613 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-103.ec2.internal" podStartSLOduration=3.235594063 podStartE2EDuration="3.235594063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:25.205825388 +0000 UTC m=+4.637460748" watchObservedRunningTime="2026-07-24 16:06:25.235594063 +0000 UTC m=+4.667229426" Jul 24 16:06:26.160794 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:26.160762 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:26.161283 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:26.160892 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:26.558221 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:26.557516 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:26.558221 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:26.557727 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:26.558221 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:26.557787 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret podName:6ea154fa-ed3f-4f3c-b3d4-980666719a5d nodeName:}" failed. No retries permitted until 2026-07-24 16:06:30.557769734 +0000 UTC m=+9.989405078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret") pod "global-pull-secret-syncer-7gtd8" (UID: "6ea154fa-ed3f-4f3c-b3d4-980666719a5d") : object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:26.658802 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:26.658761 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:26.658993 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:26.658831 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:26.658993 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:26.658987 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:26.659230 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:26.659050 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs podName:cd77de22-56d1-4095-8674-7613919037b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:30.659032276 +0000 UTC m=+10.090667617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs") pod "network-metrics-daemon-9lbnz" (UID: "cd77de22-56d1-4095-8674-7613919037b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:26.659488 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:26.659465 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 16:06:26.659564 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:26.659494 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 16:06:26.659564 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:26.659507 2560 projected.go:196] Error preparing data for projected volume kube-api-access-mhcx6 for pod openshift-network-diagnostics/network-check-target-42m7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:26.659564 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:26.659549 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6 podName:02425e0f-39d9-47b3-9a4c-c0504676aad1 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:30.659537626 +0000 UTC m=+10.091172964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhcx6" (UniqueName: "kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6") pod "network-check-target-42m7w" (UID: "02425e0f-39d9-47b3-9a4c-c0504676aad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:27.162284 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:27.162249 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:27.162743 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:27.162261 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:27.162743 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:27.162396 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:27.162743 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:27.162466 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:28.161160 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:28.161111 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:28.161328 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:28.161293 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:29.161128 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:29.161093 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:29.161575 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:29.161284 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:29.161575 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:29.161308 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:29.161575 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:29.161411 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:30.161122 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:30.161081 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:30.161361 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:30.161225 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:30.592978 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:30.592942 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:30.593204 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:30.593062 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:30.593204 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:30.593133 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret podName:6ea154fa-ed3f-4f3c-b3d4-980666719a5d nodeName:}" failed. No retries permitted until 2026-07-24 16:06:38.593116615 +0000 UTC m=+18.024751956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret") pod "global-pull-secret-syncer-7gtd8" (UID: "6ea154fa-ed3f-4f3c-b3d4-980666719a5d") : object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:30.694237 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:30.694192 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:30.694399 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:30.694250 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:30.694399 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:30.694389 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:30.694497 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:30.694452 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs podName:cd77de22-56d1-4095-8674-7613919037b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:38.694434169 +0000 UTC m=+18.126069510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs") pod "network-metrics-daemon-9lbnz" (UID: "cd77de22-56d1-4095-8674-7613919037b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:30.694899 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:30.694878 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 16:06:30.694980 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:30.694905 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 16:06:30.694980 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:30.694915 2560 projected.go:196] Error preparing data for projected volume kube-api-access-mhcx6 for pod openshift-network-diagnostics/network-check-target-42m7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:30.694980 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:30.694953 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6 podName:02425e0f-39d9-47b3-9a4c-c0504676aad1 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:38.694941568 +0000 UTC m=+18.126576906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhcx6" (UniqueName: "kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6") pod "network-check-target-42m7w" (UID: "02425e0f-39d9-47b3-9a4c-c0504676aad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:31.162463 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:31.161998 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:31.162463 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:31.162116 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:31.162463 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:31.162175 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:31.162463 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:31.162290 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:32.161350 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:32.160863 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:32.161350 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:32.160989 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:33.161315 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:33.161279 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:33.161781 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:33.161427 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:33.161781 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:33.161479 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:33.161781 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:33.161573 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:34.161089 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:34.160999 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:34.161245 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:34.161133 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:35.160836 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:35.160587 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:35.160836 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:35.160642 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:35.160836 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:35.160706 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:35.160836 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:35.160803 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:36.160475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:36.160446 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:36.160658 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:36.160569 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:37.161281 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:37.161249 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:37.161281 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:37.161268 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:37.161795 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:37.161375 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:37.161795 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:37.161507 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:38.160517 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:38.160486 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:38.160687 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:38.160586 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:38.658112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:38.658079 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:38.658565 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:38.658238 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:38.658565 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:38.658303 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret podName:6ea154fa-ed3f-4f3c-b3d4-980666719a5d nodeName:}" failed. No retries permitted until 2026-07-24 16:06:54.658284136 +0000 UTC m=+34.089919478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret") pod "global-pull-secret-syncer-7gtd8" (UID: "6ea154fa-ed3f-4f3c-b3d4-980666719a5d") : object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:38.759275 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:38.759236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:38.759446 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:38.759285 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:38.759446 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:38.759410 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:38.759446 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:38.759412 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 16:06:38.759446 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:38.759437 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 16:06:38.759446 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:38.759448 2560 projected.go:196] Error preparing data for projected volume kube-api-access-mhcx6 for pod openshift-network-diagnostics/network-check-target-42m7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:38.759684 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:38.759474 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs podName:cd77de22-56d1-4095-8674-7613919037b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:54.759454971 +0000 UTC m=+34.191090331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs") pod "network-metrics-daemon-9lbnz" (UID: "cd77de22-56d1-4095-8674-7613919037b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:38.759684 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:38.759494 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6 podName:02425e0f-39d9-47b3-9a4c-c0504676aad1 nodeName:}" failed. No retries permitted until 2026-07-24 16:06:54.759483111 +0000 UTC m=+34.191118450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhcx6" (UniqueName: "kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6") pod "network-check-target-42m7w" (UID: "02425e0f-39d9-47b3-9a4c-c0504676aad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:39.161532 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:39.161072 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:39.161532 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:39.161210 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:39.161532 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:39.161219 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:39.161532 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:39.161327 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:40.160978 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:40.160946 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:40.161442 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:40.161078 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:41.161594 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:41.161287 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:41.161594 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:41.161373 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:41.161594 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:41.161395 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:41.161594 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:41.161485 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:42.161252 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.161088 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:42.161377 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:42.161358 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:42.268391 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.268362 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" event={"ID":"a6006b0f-b1d2-47b8-9d92-0c4d60d91d88","Type":"ContainerStarted","Data":"52db5b967d1de761953c0f75efa8402046ae097977a95c779fd2604d9f1b1382"} Jul 24 16:06:42.270511 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.270488 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" event={"ID":"cc717ffd274673b36048ff8694259c2f","Type":"ContainerStarted","Data":"bfa43e5800d8a730a04ac4c34434217158576aadb49769fcbbc77868117aae53"} Jul 24 16:06:42.272112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.272087 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlv94" event={"ID":"95229c4e-2eb9-4401-8282-0cfd32c54496","Type":"ContainerStarted","Data":"cab8a0661f10f14cdd7b043a6063a3cfdfa8eac2fcf7e75c77f245b4a2416e72"} Jul 24 16:06:42.273586 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.273556 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mlv5n" event={"ID":"4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d","Type":"ContainerStarted","Data":"ee6028e9008c1bc5d26eb23f95e8e4d3582bc0748db342c260f1577e4157d944"} Jul 24 16:06:42.275042 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.275018 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" event={"ID":"665dc859-e7eb-4fa5-8756-ced75a40e249","Type":"ContainerStarted","Data":"388b79822b4606165e6ac3e7f97de05758a00c3ce62fad83372a00c9ff9c550e"} Jul 24 16:06:42.276601 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.276579 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" event={"ID":"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5","Type":"ContainerStarted","Data":"50154deb6b7f73f057bcc41e2d3942e936c8e799b6c799e10afde4082bc34f18"} Jul 24 16:06:42.278071 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.278044 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-87tpp" event={"ID":"6f746255-6327-410b-82e0-2a3b60e801fb","Type":"ContainerStarted","Data":"b213130e15d66343400b6e414f386a36ebdf36632105c0d30b701b1645835e2e"} Jul 24 16:06:42.279994 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.279963 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xwds8" event={"ID":"65297386-a516-4da5-b689-c3e03c004310","Type":"ContainerStarted","Data":"95971d8db5bcffe9bfa6a309b18f4163e94d3fd761216f7720fd25d504d148ab"} Jul 24 16:06:42.285676 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.285635 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerStarted","Data":"ada58076dc5eddf433ddc432273adb62554e1e1230f882b21e799539b758b1c5"} Jul 24 16:06:42.293829 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.293789 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4zmmf" podStartSLOduration=3.668585743 podStartE2EDuration="21.293776604s" totalImagesPullingTime="17.625190861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:42.293133305 +0000 UTC m=+21.724768662" watchObservedRunningTime="2026-07-24 16:06:42.293776604 +0000 UTC m=+21.725411961" Jul 24 16:06:42.310558 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.310109 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-103.ec2.internal" podStartSLOduration=20.310089063 podStartE2EDuration="20.310089063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:42.309707648 +0000 UTC m=+21.741343008" watchObservedRunningTime="2026-07-24 16:06:42.310089063 +0000 UTC m=+21.741724426" Jul 24 16:06:42.323836 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.323796 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-87tpp" podStartSLOduration=3.726905271 podStartE2EDuration="21.32378095s" totalImagesPullingTime="17.596875679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:42.323623836 +0000 UTC m=+21.755259223" watchObservedRunningTime="2026-07-24 16:06:42.32378095 +0000 UTC m=+21.755416309" Jul 24 16:06:42.359923 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.359876 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xwds8" podStartSLOduration=7.030737923 podStartE2EDuration="21.359860244s" totalImagesPullingTime="14.329122321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:42.359703182 +0000 UTC m=+21.791338543" watchObservedRunningTime="2026-07-24 16:06:42.359860244 +0000 UTC m=+21.791495603" Jul 24 16:06:42.378288 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.378218 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mlv5n" podStartSLOduration=2.69072918 podStartE2EDuration="21.378199607s" totalImagesPullingTime="18.687470427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:42.37816929 +0000 UTC m=+21.809804653" watchObservedRunningTime="2026-07-24 16:06:42.378199607 +0000 UTC m=+21.809834946" Jul 24 16:06:42.393640 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.393603 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hlv94" podStartSLOduration=2.756292419 podStartE2EDuration="21.393587568s" totalImagesPullingTime="18.637295149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:42.393045199 +0000 UTC m=+21.824680589" watchObservedRunningTime="2026-07-24 16:06:42.393587568 +0000 UTC m=+21.825222928" Jul 24 16:06:42.526481 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.526328 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:42.600647 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.600624 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:42.800914 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:42.800887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:43.161206 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.161173 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:43.161384 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.161177 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:43.161384 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:43.161301 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:43.161500 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:43.161397 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:43.171950 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.171922 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:43.272476 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.272401 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:43.292267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.290071 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-67vvt" event={"ID":"a8a4620c-a9c2-49f8-baa6-3806c4c9434c","Type":"ContainerStarted","Data":"649d11a2908e4be62c308789748b79546eb2ee2892a27b5efb64afc578b93340"} Jul 24 16:06:43.293547 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.293527 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:06:43.293861 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.293844 2560 generic.go:356] "Generic (PLEG): container finished" podID="b9ef7edf-3ea8-4270-b7c7-acf95e550718" containerID="7e7612f3954d2c8ea38d73a66d05f649fc30a8b2d88af214c5d111f9fee7b713" exitCode=1 Jul 24 16:06:43.293955 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.293924 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerStarted","Data":"3119b089116ffeec9fa54b5debe0bbb7e702c813e8b15acabd6218a500ea5e4a"} Jul 24 16:06:43.294000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.293987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerStarted","Data":"761d2da7039f11f4ede3219794aadc996e8128401396f756debd8d6b87821f02"} Jul 24 16:06:43.294039 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.294029 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerStarted","Data":"1e48e629cb297d4e05ca17addf7ce43e114c2dfe1216f4ffede4037d39cb694a"} Jul 24 16:06:43.294073 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.294061 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerStarted","Data":"e908e50c9da08718a29144196c0064c8d8daa300d31eef1201c2620148eb40b1"} Jul 24 16:06:43.294107 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.294092 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerDied","Data":"7e7612f3954d2c8ea38d73a66d05f649fc30a8b2d88af214c5d111f9fee7b713"} Jul 24 16:06:43.295281 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.295261 2560 generic.go:356] "Generic (PLEG): container finished" podID="c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5" containerID="50154deb6b7f73f057bcc41e2d3942e936c8e799b6c799e10afde4082bc34f18" exitCode=0 Jul 24 16:06:43.295383 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.295351 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" event={"ID":"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5","Type":"ContainerDied","Data":"50154deb6b7f73f057bcc41e2d3942e936c8e799b6c799e10afde4082bc34f18"} Jul 24 16:06:43.305602 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.305569 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-67vvt" podStartSLOduration=9.013647938 podStartE2EDuration="22.305558047s" totalImagesPullingTime="13.291910109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:43.30527816 +0000 UTC m=+22.736913523" watchObservedRunningTime="2026-07-24 16:06:43.305558047 +0000 UTC m=+22.737193406" Jul 24 16:06:43.461498 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.461468 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:43.563623 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.563597 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:43.596052 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.596031 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 16:06:43.892045 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.892025 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xwds8_65297386-a516-4da5-b689-c3e03c004310/dns-node-resolver/0.log" Jul 24 16:06:43.979538 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.979423 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T16:06:43.596047806Z","UUID":"d1557e04-b8d5-4ad8-9ed8-ed7dee59bf33","Handler":null,"Name":"","Endpoint":""} Jul 24 16:06:43.981103 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.981044 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 16:06:43.981103 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:43.981087 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 16:06:44.160438 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:44.160370 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:44.160581 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:44.160468 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:44.298698 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:44.298662 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" event={"ID":"665dc859-e7eb-4fa5-8756-ced75a40e249","Type":"ContainerStarted","Data":"a3a20165fd3b170e06dea4cba7dcbf96cc3e9db60157066ebefac55d5ff8097b"} Jul 24 16:06:44.419183 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:44.419088 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:44.477620 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:44.477590 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:44.490534 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:44.490508 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:44.491161 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:44.491123 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:44.691626 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:44.691546 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlv94_95229c4e-2eb9-4401-8282-0cfd32c54496/node-ca/0.log" Jul 24 16:06:45.160467 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:45.160440 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:45.160654 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:45.160440 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:45.160654 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:45.160603 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:45.160654 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:45.160639 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:45.262743 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:45.262486 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:45.304497 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:45.304384 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:06:45.306784 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:45.306654 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerStarted","Data":"9cd2e16e94e70c39019fed6d8d200357d11187f5666e9fc57b0be1ac29c55221"} Jul 24 16:06:45.306914 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:45.306797 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:45.307193 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:45.307175 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-87tpp" Jul 24 16:06:45.333565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:45.333535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:46.160311 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:46.160285 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:46.160595 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:46.160386 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:46.310889 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:46.310849 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" event={"ID":"665dc859-e7eb-4fa5-8756-ced75a40e249","Type":"ContainerStarted","Data":"d819889ff4df03fc25399a92af7fc61f42c0d8988f70981824c39d201c855c99"} Jul 24 16:06:46.329611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:46.329556 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hl5k8" podStartSLOduration=3.7062213269999997 podStartE2EDuration="25.329536199s" totalImagesPullingTime="21.623314872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:46.328527879 +0000 UTC m=+25.760163242" watchObservedRunningTime="2026-07-24 16:06:46.329536199 +0000 UTC m=+25.761171564" Jul 24 16:06:46.401598 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:46.401567 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:46.561039 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:46.560959 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:46.615072 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:46.615033 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:47.161235 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:47.161206 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:47.161386 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:47.161206 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:47.161386 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:47.161322 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:47.161488 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:47.161404 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:48.160960 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:48.160928 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:48.161554 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:48.161045 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:48.632215 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:48.632023 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:49.161230 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.161132 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:49.161787 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.161132 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:49.161787 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:49.161254 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:49.161787 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:49.161357 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:49.179329 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.179294 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:49.308777 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.308748 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:49.319755 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.319724 2560 generic.go:356] "Generic (PLEG): container finished" podID="c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5" containerID="39ca4793c888800db596ebac1b5abd16b26b6101248acdbd6bf9b97575363478" exitCode=0 Jul 24 16:06:49.319891 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.319802 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" event={"ID":"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5","Type":"ContainerDied","Data":"39ca4793c888800db596ebac1b5abd16b26b6101248acdbd6bf9b97575363478"} Jul 24 16:06:49.323286 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.323270 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:06:49.323640 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.323620 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerStarted","Data":"453cc7df911efc3377ca1d747315df1684f8dd0136aa8a4b5e0ca61cd382e896"} Jul 24 16:06:49.323839 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.323823 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:49.323935 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.323873 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:49.324053 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.324036 2560 scope.go:117] "RemoveContainer" containerID="7e7612f3954d2c8ea38d73a66d05f649fc30a8b2d88af214c5d111f9fee7b713" Jul 24 16:06:49.343237 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.343220 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:49.392271 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.392244 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:49.494633 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.494547 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:49.642558 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.642529 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:49.742343 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.742312 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:49.882487 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.882454 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:49.985709 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:49.985686 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:50.083902 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:50.083861 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:50.160752 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:50.160697 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:50.160834 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:50.160783 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:50.327753 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:50.327728 2560 generic.go:356] "Generic (PLEG): container finished" podID="c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5" containerID="6ca10c1484ee2af282d6d044b1a4c1225e34a4bf52f72e8f77b52e29ce6df045" exitCode=0 Jul 24 16:06:50.328190 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:50.327810 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" event={"ID":"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5","Type":"ContainerDied","Data":"6ca10c1484ee2af282d6d044b1a4c1225e34a4bf52f72e8f77b52e29ce6df045"} Jul 24 16:06:50.331538 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:50.331524 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:06:50.331844 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:50.331825 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" event={"ID":"b9ef7edf-3ea8-4270-b7c7-acf95e550718","Type":"ContainerStarted","Data":"e8be04308084e712f7f029c237befdcce3058b79f79cace663af7f53c4954c6a"} Jul 24 16:06:50.331998 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:50.331979 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:50.347282 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:50.347264 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:06:50.388027 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:50.386462 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" podStartSLOduration=11.685938206 podStartE2EDuration="29.386439694s" totalImagesPullingTime="17.700501488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:50.38621761 +0000 UTC m=+29.817852971" watchObservedRunningTime="2026-07-24 16:06:50.386439694 +0000 UTC m=+29.818075054" Jul 24 16:06:51.078436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:51.078405 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:51.163877 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:51.163807 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:51.164003 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:51.163807 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:51.164003 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:51.163914 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:51.164003 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:51.163962 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:51.336300 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:51.336272 2560 generic.go:356] "Generic (PLEG): container finished" podID="c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5" containerID="9b4f05f310962547a01f2e461e7f52bb575c5aae3b582bb62badad92ea2cabf0" exitCode=0 Jul 24 16:06:51.336728 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:51.336363 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" event={"ID":"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5","Type":"ContainerDied","Data":"9b4f05f310962547a01f2e461e7f52bb575c5aae3b582bb62badad92ea2cabf0"} Jul 24 16:06:51.623783 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:51.623762 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:51.730678 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:51.730649 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:51.878380 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:51.878304 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:52.161233 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:52.161166 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:52.161354 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:52.161255 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:52.796893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:52.796858 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:52.898232 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:52.898195 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:52.995525 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:52.995494 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:53.160435 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:53.160408 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:53.160435 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:53.160424 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:53.160677 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:53.160532 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:53.160729 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:53.160674 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:53.280605 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:53.280574 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:53.733961 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:53.733929 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:53.802842 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:53.802797 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:53.876255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:53.876213 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:54.160261 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:54.160231 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:54.160440 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:54.160336 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:54.685037 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:54.685004 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:54.685299 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:54.685175 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:54.685299 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:54.685233 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret podName:6ea154fa-ed3f-4f3c-b3d4-980666719a5d nodeName:}" failed. No retries permitted until 2026-07-24 16:07:26.685216944 +0000 UTC m=+66.116852285 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret") pod "global-pull-secret-syncer-7gtd8" (UID: "6ea154fa-ed3f-4f3c-b3d4-980666719a5d") : object "kube-system"/"original-pull-secret" not registered Jul 24 16:06:54.699607 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:54.699580 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:54.785414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:54.785380 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:54.785671 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:54.785488 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:54.785671 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:54.785554 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:54.785671 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:54.785622 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs podName:cd77de22-56d1-4095-8674-7613919037b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:07:26.785607829 +0000 UTC m=+66.217243166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs") pod "network-metrics-daemon-9lbnz" (UID: "cd77de22-56d1-4095-8674-7613919037b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 16:06:54.785671 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:54.785635 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 16:06:54.785671 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:54.785658 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 16:06:54.785671 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:54.785672 2560 projected.go:196] Error preparing data for projected volume kube-api-access-mhcx6 for pod openshift-network-diagnostics/network-check-target-42m7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:54.785910 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:54.785731 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6 podName:02425e0f-39d9-47b3-9a4c-c0504676aad1 nodeName:}" failed. No retries permitted until 2026-07-24 16:07:26.785713705 +0000 UTC m=+66.217349042 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhcx6" (UniqueName: "kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6") pod "network-check-target-42m7w" (UID: "02425e0f-39d9-47b3-9a4c-c0504676aad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 16:06:54.797671 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:54.797592 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:54.899491 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:54.899460 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:55.160870 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:55.160782 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:55.161072 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:55.160925 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:55.161072 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:55.160973 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:55.161218 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:55.161108 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:55.283037 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:55.282993 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:55.475673 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:55.475585 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:55.672587 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:55.672557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:55.891098 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:55.891074 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:56.071936 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:56.071864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:56.160698 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:56.160672 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:56.160826 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:56.160769 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:56.350131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:56.350064 2560 generic.go:356] "Generic (PLEG): container finished" podID="c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5" containerID="0fe40e3293f64b0b5bd07a24598ee7e9d9c55d70917285c2671b1f31e869c8e9" exitCode=0 Jul 24 16:06:56.350131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:56.350101 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" event={"ID":"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5","Type":"ContainerDied","Data":"0fe40e3293f64b0b5bd07a24598ee7e9d9c55d70917285c2671b1f31e869c8e9"} Jul 24 16:06:56.677438 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:56.677266 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:57.096374 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:57.096299 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:57.160846 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:57.160824 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:57.160980 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:57.160950 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:57.161022 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:57.160997 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:57.161103 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:57.161080 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:57.354757 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:57.354685 2560 generic.go:356] "Generic (PLEG): container finished" podID="c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5" containerID="01d0764c636e722b5e8086f0e498a763aa3a828fd24fc298b1641e0f674b55b2" exitCode=0 Jul 24 16:06:57.354757 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:57.354724 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" event={"ID":"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5","Type":"ContainerDied","Data":"01d0764c636e722b5e8086f0e498a763aa3a828fd24fc298b1641e0f674b55b2"} Jul 24 16:06:57.478380 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:57.478356 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:57.736805 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:57.736774 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:58.075686 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:58.075599 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:58.160925 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:58.160904 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:06:58.161363 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:58.161012 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:06:58.359475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:58.359450 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" event={"ID":"c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5","Type":"ContainerStarted","Data":"452123106e62c15d88de1bc73cc0ec6de0c0b541b40b7a922d64dc6d3603d4c9"} Jul 24 16:06:58.385842 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:58.385782 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kf2n5" podStartSLOduration=9.240987869 podStartE2EDuration="37.385767839s" totalImagesPullingTime="28.14477997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:06:58.385287109 +0000 UTC m=+37.816922469" watchObservedRunningTime="2026-07-24 16:06:58.385767839 +0000 UTC m=+37.817403198" Jul 24 16:06:58.885980 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:58.885951 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:59.160368 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:59.160303 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:06:59.160507 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:59.160429 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:06:59.160507 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:59.160478 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:06:59.160584 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:06:59.160557 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:06:59.278087 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:59.278053 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:59.477970 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:59.477895 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:59.712524 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:59.712492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:06:59.877886 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:06:59.877854 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:00.160392 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:00.160309 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:00.160557 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:00.160414 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:07:01.161849 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:01.161814 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:01.162237 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:01.161899 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:07:01.162237 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:01.162006 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:01.162237 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:01.162132 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:07:02.160777 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:02.160749 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:02.160921 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:02.160846 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:07:03.160627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:03.160588 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:03.161114 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:03.160598 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:03.161114 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:03.160696 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:07:03.161114 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:03.160775 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:07:04.160840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:04.160810 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:04.161259 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:04.160900 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:07:04.743610 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:04.743577 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:04.870771 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:04.870742 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:04.973588 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:04.973563 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:05.092473 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:05.092412 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:05.161079 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:05.161059 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:05.161436 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:05.161205 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:07:05.161436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:05.161261 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:05.161436 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:05.161338 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:07:05.560917 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:05.560813 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:05.626577 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:05.626534 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:06.160771 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:06.160742 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:06.160985 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:06.160857 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:07:06.533610 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:06.533225 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7gtd8"] Jul 24 16:07:06.533610 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:06.533345 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:06.533610 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:06.533457 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:07:06.538658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:06.538295 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9lbnz"] Jul 24 16:07:06.538658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:06.538395 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:06.538658 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:06.538489 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:07:06.539116 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:06.539098 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-42m7w"] Jul 24 16:07:06.539237 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:06.539201 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:06.539316 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:06.539292 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:07:06.841794 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:06.841726 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:06.915979 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:06.915950 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:08.160939 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:08.160910 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:08.160939 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:08.160929 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:08.161399 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:08.160910 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:08.161399 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:08.161002 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:07:08.161399 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:08.161099 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:07:08.161399 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:08.161170 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:07:10.161314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:10.161283 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:10.161690 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:10.161369 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:10.161690 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:10.161379 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:10.161690 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:10.161372 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42m7w" podUID="02425e0f-39d9-47b3-9a4c-c0504676aad1" Jul 24 16:07:10.161690 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:10.161476 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gtd8" podUID="6ea154fa-ed3f-4f3c-b3d4-980666719a5d" Jul 24 16:07:10.161690 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:10.161536 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9lbnz" podUID="cd77de22-56d1-4095-8674-7613919037b0" Jul 24 16:07:10.907576 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:10.907406 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-103.ec2.internal" event="NodeReady" Jul 24 16:07:10.907790 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:10.907653 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 16:07:11.014100 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.014063 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pmbd4"] Jul 24 16:07:11.052431 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.052401 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qr5k7"] Jul 24 16:07:11.052609 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.052517 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.058440 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.058417 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 16:07:11.058668 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.058563 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v6rwh\"" Jul 24 16:07:11.058793 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.058699 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 16:07:11.076676 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.076654 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pmbd4"] Jul 24 16:07:11.076759 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.076685 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qr5k7"] Jul 24 16:07:11.076797 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.076756 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qr5k7" Jul 24 16:07:11.079661 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.079647 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 16:07:11.080537 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.080523 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 16:07:11.081232 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.081213 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7xbff\"" Jul 24 16:07:11.081743 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.081729 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 16:07:11.106136 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.106116 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ad1888-b16b-4aee-9925-95a654c4ff65-config-volume\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.106246 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.106163 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1888-b16b-4aee-9925-95a654c4ff65-tmp-dir\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.106246 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.106202 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6ad1888-b16b-4aee-9925-95a654c4ff65-metrics-tls\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.106246 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.106228 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a059404-9f78-4a76-a3d0-38bdb8969166-cert\") pod \"ingress-canary-qr5k7\" (UID: \"6a059404-9f78-4a76-a3d0-38bdb8969166\") " pod="openshift-ingress-canary/ingress-canary-qr5k7" Jul 24 16:07:11.106358 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.106254 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnv8s\" (UniqueName: \"kubernetes.io/projected/e6ad1888-b16b-4aee-9925-95a654c4ff65-kube-api-access-nnv8s\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.106358 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.106271 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrcl\" (UniqueName: \"kubernetes.io/projected/6a059404-9f78-4a76-a3d0-38bdb8969166-kube-api-access-hcrcl\") pod \"ingress-canary-qr5k7\" (UID: \"6a059404-9f78-4a76-a3d0-38bdb8969166\") " pod="openshift-ingress-canary/ingress-canary-qr5k7" Jul 24 16:07:11.108459 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.108444 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x2mw4"] Jul 24 16:07:11.133156 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.133127 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.133416 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.133382 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x2mw4"] Jul 24 16:07:11.136791 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.136772 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 16:07:11.136959 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.136942 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 16:07:11.137168 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.137133 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 16:07:11.137556 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.137538 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l4gkx\"" Jul 24 16:07:11.137647 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.137564 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 16:07:11.207495 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6ad1888-b16b-4aee-9925-95a654c4ff65-metrics-tls\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.207495 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a059404-9f78-4a76-a3d0-38bdb8969166-cert\") pod \"ingress-canary-qr5k7\" (UID: \"6a059404-9f78-4a76-a3d0-38bdb8969166\") " pod="openshift-ingress-canary/ingress-canary-qr5k7" Jul 24 16:07:11.207495 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207486 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnv8s\" (UniqueName: \"kubernetes.io/projected/e6ad1888-b16b-4aee-9925-95a654c4ff65-kube-api-access-nnv8s\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.207933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207511 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrcl\" (UniqueName: \"kubernetes.io/projected/6a059404-9f78-4a76-a3d0-38bdb8969166-kube-api-access-hcrcl\") pod \"ingress-canary-qr5k7\" (UID: \"6a059404-9f78-4a76-a3d0-38bdb8969166\") " pod="openshift-ingress-canary/ingress-canary-qr5k7" Jul 24 16:07:11.207933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207559 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.207933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207583 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.207933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207633 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ad1888-b16b-4aee-9925-95a654c4ff65-config-volume\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.207933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207675 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1888-b16b-4aee-9925-95a654c4ff65-tmp-dir\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.207933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207815 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-data-volume\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.207933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207845 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmpkz\" (UniqueName: \"kubernetes.io/projected/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-kube-api-access-bmpkz\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.207933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.207885 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-crio-socket\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.208351 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.208067 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ad1888-b16b-4aee-9925-95a654c4ff65-tmp-dir\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.208351 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.208319 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ad1888-b16b-4aee-9925-95a654c4ff65-config-volume\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.211523 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.211501 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6ad1888-b16b-4aee-9925-95a654c4ff65-metrics-tls\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.211609 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.211521 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a059404-9f78-4a76-a3d0-38bdb8969166-cert\") pod \"ingress-canary-qr5k7\" (UID: \"6a059404-9f78-4a76-a3d0-38bdb8969166\") " pod="openshift-ingress-canary/ingress-canary-qr5k7" Jul 24 16:07:11.225560 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.225538 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnv8s\" (UniqueName: \"kubernetes.io/projected/e6ad1888-b16b-4aee-9925-95a654c4ff65-kube-api-access-nnv8s\") pod \"dns-default-pmbd4\" (UID: \"e6ad1888-b16b-4aee-9925-95a654c4ff65\") " pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.228016 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.227991 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrcl\" (UniqueName: \"kubernetes.io/projected/6a059404-9f78-4a76-a3d0-38bdb8969166-kube-api-access-hcrcl\") pod \"ingress-canary-qr5k7\" (UID: \"6a059404-9f78-4a76-a3d0-38bdb8969166\") " pod="openshift-ingress-canary/ingress-canary-qr5k7" Jul 24 16:07:11.262345 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.262318 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:11.308492 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.308467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.308584 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.308499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.308584 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.308558 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-data-volume\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.308684 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.308581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmpkz\" (UniqueName: \"kubernetes.io/projected/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-kube-api-access-bmpkz\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.308684 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.308606 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-crio-socket\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.308800 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.308785 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-crio-socket\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.308844 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.308814 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-data-volume\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.308964 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.308947 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.310811 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.310789 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.316844 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.316828 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmpkz\" (UniqueName: \"kubernetes.io/projected/a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d-kube-api-access-bmpkz\") pod \"insights-runtime-extractor-x2mw4\" (UID: \"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d\") " pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.359620 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.359593 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:11.364487 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.364475 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:11.384036 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.384019 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qr5k7" Jul 24 16:07:11.443299 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.443272 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x2mw4" Jul 24 16:07:11.579790 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.579762 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qr5k7"] Jul 24 16:07:11.582886 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:11.582834 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a059404_9f78_4a76_a3d0_38bdb8969166.slice/crio-b736a5dd4df1952253528dfe082719f8bcaf394262d82b0c411b892ee81dc126 WatchSource:0}: Error finding container b736a5dd4df1952253528dfe082719f8bcaf394262d82b0c411b892ee81dc126: Status 404 returned error can't find the container with id b736a5dd4df1952253528dfe082719f8bcaf394262d82b0c411b892ee81dc126 Jul 24 16:07:11.584312 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.584293 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pmbd4"] Jul 24 16:07:11.587607 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:11.587585 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ad1888_b16b_4aee_9925_95a654c4ff65.slice/crio-9e919e826465a69909b50f491233adbb968c3efc8581d12d555bfdade70f1371 WatchSource:0}: Error finding container 9e919e826465a69909b50f491233adbb968c3efc8581d12d555bfdade70f1371: Status 404 returned error can't find the container with id 9e919e826465a69909b50f491233adbb968c3efc8581d12d555bfdade70f1371 Jul 24 16:07:11.597368 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:11.597349 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x2mw4"] Jul 24 16:07:11.600681 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:11.600663 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5022dcf_82e1_4dca_bcd0_e1fdd8aacf6d.slice/crio-c23f238566284ecad63b2c73502eb7840cb89c891fdf5233375e0ab24a99851d WatchSource:0}: Error finding container c23f238566284ecad63b2c73502eb7840cb89c891fdf5233375e0ab24a99851d: Status 404 returned error can't find the container with id c23f238566284ecad63b2c73502eb7840cb89c891fdf5233375e0ab24a99851d Jul 24 16:07:12.107778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.107730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:12.161112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.161068 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:12.161544 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.161507 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:12.162277 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.161856 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:12.164546 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.164348 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2nb27\"" Jul 24 16:07:12.165578 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.165558 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 16:07:12.165716 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.165686 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 16:07:12.165925 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.165783 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 16:07:12.165925 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.165836 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fvsfq\"" Jul 24 16:07:12.165925 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.165897 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 16:07:12.224177 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.223344 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-xmnf6"] Jul 24 16:07:12.224988 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.224963 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:12.237192 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.237174 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xmnf6" Jul 24 16:07:12.242738 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.242715 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xmnf6"] Jul 24 16:07:12.243592 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.243242 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 16:07:12.243592 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.243447 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 16:07:12.243592 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.243539 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5whrf\"" Jul 24 16:07:12.316543 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.316456 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mcf\" (UniqueName: \"kubernetes.io/projected/e514e66f-f0cb-4119-b3a2-555d1d6702be-kube-api-access-r6mcf\") pod \"downloads-7c97dc4555-xmnf6\" (UID: \"e514e66f-f0cb-4119-b3a2-555d1d6702be\") " pod="openshift-console/downloads-7c97dc4555-xmnf6" Jul 24 16:07:12.319606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.319551 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:12.349514 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.349484 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6"] Jul 24 16:07:12.369958 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.369897 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" Jul 24 16:07:12.369958 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.369940 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6"] Jul 24 16:07:12.372266 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.372186 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 24 16:07:12.372390 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.372272 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nd42l\"" Jul 24 16:07:12.393481 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.393457 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:12.394232 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.394206 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qr5k7" event={"ID":"6a059404-9f78-4a76-a3d0-38bdb8969166","Type":"ContainerStarted","Data":"b736a5dd4df1952253528dfe082719f8bcaf394262d82b0c411b892ee81dc126"} Jul 24 16:07:12.395531 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.395509 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2mw4" event={"ID":"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d","Type":"ContainerStarted","Data":"c23f238566284ecad63b2c73502eb7840cb89c891fdf5233375e0ab24a99851d"} Jul 24 16:07:12.396749 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.396731 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pmbd4" event={"ID":"e6ad1888-b16b-4aee-9925-95a654c4ff65","Type":"ContainerStarted","Data":"9e919e826465a69909b50f491233adbb968c3efc8581d12d555bfdade70f1371"} Jul 24 16:07:12.417384 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.417170 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eab9cdfe-4b71-428f-b037-7b85300f7f30-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-zlgc6\" (UID: \"eab9cdfe-4b71-428f-b037-7b85300f7f30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" Jul 24 16:07:12.417384 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.417282 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mcf\" (UniqueName: \"kubernetes.io/projected/e514e66f-f0cb-4119-b3a2-555d1d6702be-kube-api-access-r6mcf\") pod \"downloads-7c97dc4555-xmnf6\" (UID: \"e514e66f-f0cb-4119-b3a2-555d1d6702be\") " pod="openshift-console/downloads-7c97dc4555-xmnf6" Jul 24 16:07:12.439363 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.439310 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mcf\" (UniqueName: \"kubernetes.io/projected/e514e66f-f0cb-4119-b3a2-555d1d6702be-kube-api-access-r6mcf\") pod \"downloads-7c97dc4555-xmnf6\" (UID: \"e514e66f-f0cb-4119-b3a2-555d1d6702be\") " pod="openshift-console/downloads-7c97dc4555-xmnf6" Jul 24 16:07:12.518067 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.517990 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eab9cdfe-4b71-428f-b037-7b85300f7f30-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-zlgc6\" (UID: \"eab9cdfe-4b71-428f-b037-7b85300f7f30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" Jul 24 16:07:12.518836 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:12.518490 2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 24 16:07:12.518836 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:12.518600 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eab9cdfe-4b71-428f-b037-7b85300f7f30-tls-certificates podName:eab9cdfe-4b71-428f-b037-7b85300f7f30 nodeName:}" failed. No retries permitted until 2026-07-24 16:07:13.018578098 +0000 UTC m=+52.450213452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/eab9cdfe-4b71-428f-b037-7b85300f7f30-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" (UID: "eab9cdfe-4b71-428f-b037-7b85300f7f30") : secret "prometheus-operator-admission-webhook-tls" not found Jul 24 16:07:12.553508 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.553231 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xmnf6" Jul 24 16:07:12.651653 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.651135 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:12.753203 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.752939 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xmnf6"] Jul 24 16:07:12.803002 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:12.802968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:13.023538 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.023460 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eab9cdfe-4b71-428f-b037-7b85300f7f30-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-zlgc6\" (UID: \"eab9cdfe-4b71-428f-b037-7b85300f7f30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" Jul 24 16:07:13.027649 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.027595 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eab9cdfe-4b71-428f-b037-7b85300f7f30-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-zlgc6\" (UID: \"eab9cdfe-4b71-428f-b037-7b85300f7f30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" Jul 24 16:07:13.246467 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.246336 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:13.283583 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.283259 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" Jul 24 16:07:13.351714 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.351579 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:13.402449 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.402357 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xmnf6" event={"ID":"e514e66f-f0cb-4119-b3a2-555d1d6702be","Type":"ContainerStarted","Data":"ffd5f3137f53c60c6c2a6bb5a3c0a308acbc369e726a4646e1b7d72d5b2f6191"} Jul 24 16:07:13.448439 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.448263 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:13.643655 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.643567 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:13.705372 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.705101 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:13.889389 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.889350 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:13.995820 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:13.995739 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:14.097675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:14.097561 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:14.288086 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:14.288014 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:14.406278 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:14.406195 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:15.405217 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:15.405185 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:15.505064 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:15.505032 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:15.581710 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:15.581679 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:15.855060 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:15.854952 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:15.955115 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:15.955070 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:17.409203 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.409176 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fcb847f95-h2w7j"] Jul 24 16:07:17.603272 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.603240 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fcb847f95-h2w7j"] Jul 24 16:07:17.603445 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.603377 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.607023 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.607000 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 16:07:17.607023 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.607013 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 16:07:17.607264 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.607249 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 16:07:17.607330 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.607298 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 16:07:17.607961 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.607937 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lfmc9\"" Jul 24 16:07:17.608176 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.608136 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 16:07:17.668688 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.668553 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-serving-cert\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.668688 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.668618 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-console-config\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.668688 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.668656 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsmcq\" (UniqueName: \"kubernetes.io/projected/727735e9-6b62-45d0-adca-6b988fe783b9-kube-api-access-fsmcq\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.668971 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.668751 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-oauth-config\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.668971 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.668787 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-oauth-serving-cert\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.668971 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.668824 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-service-ca\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.769580 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.769544 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-serving-cert\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.769762 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.769607 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-console-config\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.769762 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.769644 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsmcq\" (UniqueName: \"kubernetes.io/projected/727735e9-6b62-45d0-adca-6b988fe783b9-kube-api-access-fsmcq\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.769881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.769801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-oauth-config\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.769881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.769834 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-oauth-serving-cert\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.769881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.769866 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-service-ca\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.770922 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.770893 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-oauth-serving-cert\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.771030 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.770935 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-service-ca\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.771030 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.770969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-console-config\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.773017 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.772975 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-serving-cert\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.773097 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.773061 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-oauth-config\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.778876 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.778844 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsmcq\" (UniqueName: \"kubernetes.io/projected/727735e9-6b62-45d0-adca-6b988fe783b9-kube-api-access-fsmcq\") pod \"console-6fcb847f95-h2w7j\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:17.917164 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:17.917112 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:18.807004 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:18.806968 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6"] Jul 24 16:07:19.123072 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:19.123045 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab9cdfe_4b71_428f_b037_7b85300f7f30.slice/crio-01fe0c158c8bd4a393367d263f6c4c8fbed61ed4ea252ecc50c04eda44b54666 WatchSource:0}: Error finding container 01fe0c158c8bd4a393367d263f6c4c8fbed61ed4ea252ecc50c04eda44b54666: Status 404 returned error can't find the container with id 01fe0c158c8bd4a393367d263f6c4c8fbed61ed4ea252ecc50c04eda44b54666 Jul 24 16:07:19.278441 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.278314 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fcb847f95-h2w7j"] Jul 24 16:07:19.284988 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:19.284649 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727735e9_6b62_45d0_adca_6b988fe783b9.slice/crio-370fcaff425af3e80ce16cf1cac2236d2ed4c289964e9b8ba36966a9b22c80f8 WatchSource:0}: Error finding container 370fcaff425af3e80ce16cf1cac2236d2ed4c289964e9b8ba36966a9b22c80f8: Status 404 returned error can't find the container with id 370fcaff425af3e80ce16cf1cac2236d2ed4c289964e9b8ba36966a9b22c80f8 Jul 24 16:07:19.421599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.421469 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2mw4" event={"ID":"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d","Type":"ContainerStarted","Data":"631e18c1b0e65e4d5258842ad71f53244b18aff8cb8eccca49df401b3aab8a2d"} Jul 24 16:07:19.424041 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.423324 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pmbd4" event={"ID":"e6ad1888-b16b-4aee-9925-95a654c4ff65","Type":"ContainerStarted","Data":"951eb89d3ba37b0b5b48829ba1d4324284f9b8b32fcc60f58b8b4143ac443b3b"} Jul 24 16:07:19.424041 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.423366 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pmbd4" event={"ID":"e6ad1888-b16b-4aee-9925-95a654c4ff65","Type":"ContainerStarted","Data":"ef70cc609f447b5b8f676aad02860bdbd5f8d158d17cb94cbeff1aa2ab113cfb"} Jul 24 16:07:19.424041 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.423991 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:19.425781 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.425467 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qr5k7" event={"ID":"6a059404-9f78-4a76-a3d0-38bdb8969166","Type":"ContainerStarted","Data":"df476d846fb828ccb34563d462e9b280598fd36a1fd7636a03040669fb2db6f6"} Jul 24 16:07:19.427026 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.426986 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fcb847f95-h2w7j" event={"ID":"727735e9-6b62-45d0-adca-6b988fe783b9","Type":"ContainerStarted","Data":"370fcaff425af3e80ce16cf1cac2236d2ed4c289964e9b8ba36966a9b22c80f8"} Jul 24 16:07:19.428414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.428393 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" event={"ID":"eab9cdfe-4b71-428f-b037-7b85300f7f30","Type":"ContainerStarted","Data":"01fe0c158c8bd4a393367d263f6c4c8fbed61ed4ea252ecc50c04eda44b54666"} Jul 24 16:07:19.454605 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.454554 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pmbd4" podStartSLOduration=1.9237056670000001 podStartE2EDuration="9.454539492s" totalImagesPullingTime="7.530833825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:19.453854656 +0000 UTC m=+58.885490018" watchObservedRunningTime="2026-07-24 16:07:19.454539492 +0000 UTC m=+58.886174843" Jul 24 16:07:19.482401 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:19.482346 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qr5k7" podStartSLOduration=1.9479155929999998 podStartE2EDuration="9.48233169s" totalImagesPullingTime="7.534416097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:19.481718333 +0000 UTC m=+58.913353694" watchObservedRunningTime="2026-07-24 16:07:19.48233169 +0000 UTC m=+58.913967052" Jul 24 16:07:20.441180 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:20.441076 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" event={"ID":"eab9cdfe-4b71-428f-b037-7b85300f7f30","Type":"ContainerStarted","Data":"4f61f153469ce8a3f1489b706d1e568ab38ccaae92b8f0522090906febb85fdd"} Jul 24 16:07:20.441628 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:20.441471 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" Jul 24 16:07:20.448171 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:20.448107 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" Jul 24 16:07:20.458058 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:20.458018 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zlgc6" podStartSLOduration=7.407825256 podStartE2EDuration="8.458003462s" totalImagesPullingTime="1.050178206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:20.457675442 +0000 UTC m=+59.889310803" watchObservedRunningTime="2026-07-24 16:07:20.458003462 +0000 UTC m=+59.889638804" Jul 24 16:07:20.898971 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:20.898910 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:21.037976 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.037908 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:21.378968 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.378916 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:21.418168 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.418112 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-r2qp6"] Jul 24 16:07:21.420429 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.420409 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.423780 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.423757 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 24 16:07:21.423987 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.423971 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 16:07:21.424237 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.424202 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 16:07:21.424386 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.424348 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8hqzm\"" Jul 24 16:07:21.424553 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.424529 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 16:07:21.424742 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.424725 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 24 16:07:21.442349 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.442328 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-r2qp6"] Jul 24 16:07:21.465001 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.464974 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:21.501042 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.501010 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bd7f79f-23d4-41f9-afbc-c6ee87318297-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.501207 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.501061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bd7f79f-23d4-41f9-afbc-c6ee87318297-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.501207 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.501093 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p26gf\" (UniqueName: \"kubernetes.io/projected/9bd7f79f-23d4-41f9-afbc-c6ee87318297-kube-api-access-p26gf\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.501207 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.501191 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bd7f79f-23d4-41f9-afbc-c6ee87318297-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.536374 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.536327 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:21.603506 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.602100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bd7f79f-23d4-41f9-afbc-c6ee87318297-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.603506 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.602240 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bd7f79f-23d4-41f9-afbc-c6ee87318297-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.603506 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.602273 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bd7f79f-23d4-41f9-afbc-c6ee87318297-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.603506 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.602302 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p26gf\" (UniqueName: \"kubernetes.io/projected/9bd7f79f-23d4-41f9-afbc-c6ee87318297-kube-api-access-p26gf\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.603976 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.603923 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bd7f79f-23d4-41f9-afbc-c6ee87318297-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.607225 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.607183 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bd7f79f-23d4-41f9-afbc-c6ee87318297-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.609256 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.609216 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bd7f79f-23d4-41f9-afbc-c6ee87318297-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.611824 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.611783 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p26gf\" (UniqueName: \"kubernetes.io/projected/9bd7f79f-23d4-41f9-afbc-c6ee87318297-kube-api-access-p26gf\") pod \"prometheus-operator-84d69f794d-r2qp6\" (UID: \"9bd7f79f-23d4-41f9-afbc-c6ee87318297\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:21.734335 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:21.734248 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" Jul 24 16:07:22.351459 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:22.351424 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlf24" Jul 24 16:07:22.492779 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:22.492745 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:22.551681 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:22.551640 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:24.509549 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:24.509510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:24.569852 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:24.569820 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:25.299975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.299854 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:25.300248 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.300225 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-r2qp6"] Jul 24 16:07:25.302328 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:25.302300 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd7f79f_23d4_41f9_afbc_c6ee87318297.slice/crio-c023b62b5cccc7d0e3d87f7af3e5f412bd35167b03900432a48373891238fda0 WatchSource:0}: Error finding container c023b62b5cccc7d0e3d87f7af3e5f412bd35167b03900432a48373891238fda0: Status 404 returned error can't find the container with id c023b62b5cccc7d0e3d87f7af3e5f412bd35167b03900432a48373891238fda0 Jul 24 16:07:25.394524 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.394439 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:25.461565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.461526 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" event={"ID":"9bd7f79f-23d4-41f9-afbc-c6ee87318297","Type":"ContainerStarted","Data":"c023b62b5cccc7d0e3d87f7af3e5f412bd35167b03900432a48373891238fda0"} Jul 24 16:07:25.463251 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.463222 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fcb847f95-h2w7j" event={"ID":"727735e9-6b62-45d0-adca-6b988fe783b9","Type":"ContainerStarted","Data":"4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0"} Jul 24 16:07:25.465304 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.465279 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2mw4" event={"ID":"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d","Type":"ContainerStarted","Data":"801b4adcdd8f2f27354f99e46e7fbe55bf9ca56013ce18e7d8400d253b0b3035"} Jul 24 16:07:25.481666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.481620 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fcb847f95-h2w7j" podStartSLOduration=2.631010182 podStartE2EDuration="8.481604538s" totalImagesPullingTime="5.850594356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:25.480451026 +0000 UTC m=+64.912086397" watchObservedRunningTime="2026-07-24 16:07:25.481604538 +0000 UTC m=+64.913239897" Jul 24 16:07:25.898395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.898357 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-87f758789-jc6mj"] Jul 24 16:07:25.901456 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.901431 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:25.910675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.910650 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 16:07:25.917369 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:25.917312 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87f758789-jc6mj"] Jul 24 16:07:26.043633 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.043596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kfx4\" (UniqueName: \"kubernetes.io/projected/61cbab77-e0a5-48d3-97bc-7e3af545a45b-kube-api-access-5kfx4\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.043633 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.043646 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-oauth-config\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.043873 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.043704 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-oauth-serving-cert\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.043873 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.043737 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-serving-cert\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.043873 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.043783 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-config\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.043873 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.043809 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-trusted-ca-bundle\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.043873 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.043846 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-service-ca\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.145099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.145064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kfx4\" (UniqueName: \"kubernetes.io/projected/61cbab77-e0a5-48d3-97bc-7e3af545a45b-kube-api-access-5kfx4\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.145274 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.145121 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-oauth-config\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.145274 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.145203 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-oauth-serving-cert\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.145274 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.145260 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-serving-cert\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.145449 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.145300 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-config\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.145449 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.145430 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-trusted-ca-bundle\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.145550 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.145483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-service-ca\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.146020 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.145995 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-oauth-serving-cert\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.146255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.146210 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-config\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.146567 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.146539 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-trusted-ca-bundle\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.147063 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.147041 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-service-ca\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.148132 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.148114 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-oauth-config\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.148336 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.148316 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-serving-cert\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.156264 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.156207 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kfx4\" (UniqueName: \"kubernetes.io/projected/61cbab77-e0a5-48d3-97bc-7e3af545a45b-kube-api-access-5kfx4\") pod \"console-87f758789-jc6mj\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.212855 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.212824 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:26.737835 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.737789 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:26.750626 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.750593 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:26.753554 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.753526 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 16:07:26.763771 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.763750 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ea154fa-ed3f-4f3c-b3d4-980666719a5d-original-pull-secret\") pod \"global-pull-secret-syncer-7gtd8\" (UID: \"6ea154fa-ed3f-4f3c-b3d4-980666719a5d\") " pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:26.843230 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.843202 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:26.851280 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.851248 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:26.851408 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.851317 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:26.853984 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.853965 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 16:07:26.854658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.854629 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 16:07:26.863974 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.863955 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 16:07:26.864162 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.864132 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd77de22-56d1-4095-8674-7613919037b0-metrics-certs\") pod \"network-metrics-daemon-9lbnz\" (UID: \"cd77de22-56d1-4095-8674-7613919037b0\") " pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:26.874706 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.874683 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcx6\" (UniqueName: \"kubernetes.io/projected/02425e0f-39d9-47b3-9a4c-c0504676aad1-kube-api-access-mhcx6\") pod \"network-check-target-42m7w\" (UID: \"02425e0f-39d9-47b3-9a4c-c0504676aad1\") " pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:26.878025 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.878005 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2nb27\"" Jul 24 16:07:26.886394 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.886374 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:26.886673 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.886657 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fvsfq\"" Jul 24 16:07:26.892017 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.892001 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gtd8" Jul 24 16:07:26.894687 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:26.894672 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9lbnz" Jul 24 16:07:27.917420 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:27.917384 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:27.917855 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:27.917451 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:27.922888 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:27.922864 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:28.480117 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:28.480086 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:07:29.731076 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:29.731043 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:29.787477 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:29.787445 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:29.955860 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:29.955826 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:30.062704 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:30.062628 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:30.154549 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:30.154517 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:30.254083 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:30.254044 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:30.353771 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:30.353641 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:30.448693 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:30.448667 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pmbd4" Jul 24 16:07:32.504587 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:32.504515 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7gtd8"] Jul 24 16:07:32.538003 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:32.537917 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87f758789-jc6mj"] Jul 24 16:07:32.577864 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:32.577827 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea154fa_ed3f_4f3c_b3d4_980666719a5d.slice/crio-e3ee6b4aee5f2592abb8e7c2301efcf847d81e5cbbd3734a5c4ff807f334db4f WatchSource:0}: Error finding container e3ee6b4aee5f2592abb8e7c2301efcf847d81e5cbbd3734a5c4ff807f334db4f: Status 404 returned error can't find the container with id e3ee6b4aee5f2592abb8e7c2301efcf847d81e5cbbd3734a5c4ff807f334db4f Jul 24 16:07:32.578741 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:32.578706 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cbab77_e0a5_48d3_97bc_7e3af545a45b.slice/crio-cbdf57abe4715f87bf8333a87011d5df91de6d7c973f5594e7a9f768d591514d WatchSource:0}: Error finding container cbdf57abe4715f87bf8333a87011d5df91de6d7c973f5594e7a9f768d591514d: Status 404 returned error can't find the container with id cbdf57abe4715f87bf8333a87011d5df91de6d7c973f5594e7a9f768d591514d Jul 24 16:07:32.759401 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:32.759362 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9lbnz"] Jul 24 16:07:32.765736 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:32.765703 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd77de22_56d1_4095_8674_7613919037b0.slice/crio-3f927767ae3adb5a701eb3d9e772cb8a404052e7740c8aa2c4c2984d2cb13fa1 WatchSource:0}: Error finding container 3f927767ae3adb5a701eb3d9e772cb8a404052e7740c8aa2c4c2984d2cb13fa1: Status 404 returned error can't find the container with id 3f927767ae3adb5a701eb3d9e772cb8a404052e7740c8aa2c4c2984d2cb13fa1 Jul 24 16:07:32.766632 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:32.766598 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-42m7w"] Jul 24 16:07:33.511274 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.510231 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xmnf6" event={"ID":"e514e66f-f0cb-4119-b3a2-555d1d6702be","Type":"ContainerStarted","Data":"d294750227598919cc1d9f438d05335c56b7ef3b254863c125da185079380afa"} Jul 24 16:07:33.512052 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.511994 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-xmnf6" Jul 24 16:07:33.514134 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.514080 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-42m7w" event={"ID":"02425e0f-39d9-47b3-9a4c-c0504676aad1","Type":"ContainerStarted","Data":"1872020e4e662b0c05b717a9f261ec8ecc98e94e891490c41374bad3a5f7cc64"} Jul 24 16:07:33.515490 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.515435 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7gtd8" event={"ID":"6ea154fa-ed3f-4f3c-b3d4-980666719a5d","Type":"ContainerStarted","Data":"e3ee6b4aee5f2592abb8e7c2301efcf847d81e5cbbd3734a5c4ff807f334db4f"} Jul 24 16:07:33.518020 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.517994 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9lbnz" event={"ID":"cd77de22-56d1-4095-8674-7613919037b0","Type":"ContainerStarted","Data":"3f927767ae3adb5a701eb3d9e772cb8a404052e7740c8aa2c4c2984d2cb13fa1"} Jul 24 16:07:33.520590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.520563 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87f758789-jc6mj" event={"ID":"61cbab77-e0a5-48d3-97bc-7e3af545a45b","Type":"ContainerStarted","Data":"60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf"} Jul 24 16:07:33.520686 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.520648 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87f758789-jc6mj" event={"ID":"61cbab77-e0a5-48d3-97bc-7e3af545a45b","Type":"ContainerStarted","Data":"cbdf57abe4715f87bf8333a87011d5df91de6d7c973f5594e7a9f768d591514d"} Jul 24 16:07:33.534704 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.534428 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-xmnf6" podStartSLOduration=1.625378089 podStartE2EDuration="21.534407981s" totalImagesPullingTime="19.909029892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:33.533604117 +0000 UTC m=+72.965239481" watchObservedRunningTime="2026-07-24 16:07:33.534407981 +0000 UTC m=+72.966043321" Jul 24 16:07:33.536841 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.536796 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-xmnf6" Jul 24 16:07:33.557046 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:33.556908 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-87f758789-jc6mj" podStartSLOduration=8.556885194 podStartE2EDuration="8.556885194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:33.556292136 +0000 UTC m=+72.987927509" watchObservedRunningTime="2026-07-24 16:07:33.556885194 +0000 UTC m=+72.988520534" Jul 24 16:07:34.118103 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:34.118052 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:34.212844 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:34.212672 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:34.963810 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:34.963778 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:35.074804 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:35.074711 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:35.168579 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:35.168505 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:35.538460 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:35.538422 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" event={"ID":"9bd7f79f-23d4-41f9-afbc-c6ee87318297","Type":"ContainerStarted","Data":"cfe50317c335499d273bc511b1690e3fc37747df0acfe243929146ca987d05c6"} Jul 24 16:07:35.807787 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:35.807720 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:36.214057 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:36.213602 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:36.214057 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:36.213673 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:36.220673 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:36.220441 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:36.550280 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:36.550203 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9lbnz" event={"ID":"cd77de22-56d1-4095-8674-7613919037b0","Type":"ContainerStarted","Data":"5ea5978acfd3baf230405d0b53c383b1913f89042923d7579ba578e722b1b70c"} Jul 24 16:07:36.556520 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:36.556317 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:07:36.610641 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:36.609917 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fcb847f95-h2w7j"] Jul 24 16:07:37.139585 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:37.139215 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:37.238375 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:37.238282 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:38.432452 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:38.432390 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:39.582282 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:39.582244 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:39.646888 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:39.646845 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:43.579591 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.579558 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-42m7w" event={"ID":"02425e0f-39d9-47b3-9a4c-c0504676aad1","Type":"ContainerStarted","Data":"81cb9b643d8fdaaae0b1e9ee6d21c4cab300de758d9d16001671e3b652cf4cb0"} Jul 24 16:07:43.580033 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.579684 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:07:43.581873 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.581841 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2mw4" event={"ID":"a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d","Type":"ContainerStarted","Data":"1e15a20b222fcf095a1526ad1e6561d6c1d858389af94a1e315ea7961405bb06"} Jul 24 16:07:43.583368 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.583345 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7gtd8" event={"ID":"6ea154fa-ed3f-4f3c-b3d4-980666719a5d","Type":"ContainerStarted","Data":"68044c17fc0c5db75d8180b995fb36b4062c58ed7951fa23d2a85a52583aa253"} Jul 24 16:07:43.585055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.585030 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9lbnz" event={"ID":"cd77de22-56d1-4095-8674-7613919037b0","Type":"ContainerStarted","Data":"f49d22d498db1269abc0c657882460ab4ccf8221fed69b1c10f942838247b75f"} Jul 24 16:07:43.586750 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.586730 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" event={"ID":"9bd7f79f-23d4-41f9-afbc-c6ee87318297","Type":"ContainerStarted","Data":"f6405823780972196d0792d96d6809047c1ffbfdf932aac1592695c5a965dbd7"} Jul 24 16:07:43.597275 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.597231 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-42m7w" podStartSLOduration=72.377106172 podStartE2EDuration="1m22.597215324s" totalImagesPullingTime="10.220109152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:43.596641662 +0000 UTC m=+83.028277026" watchObservedRunningTime="2026-07-24 16:07:43.597215324 +0000 UTC m=+83.028850679" Jul 24 16:07:43.618211 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.618024 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x2mw4" podStartSLOduration=1.372582516 podStartE2EDuration="32.618006228s" totalImagesPullingTime="31.245423712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:07:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:43.61706513 +0000 UTC m=+83.048700501" watchObservedRunningTime="2026-07-24 16:07:43.618006228 +0000 UTC m=+83.049641589" Jul 24 16:07:43.632822 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.632746 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9lbnz" podStartSLOduration=80.091705514 podStartE2EDuration="1m22.632727311s" totalImagesPullingTime="2.541021797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:43.632299777 +0000 UTC m=+83.063935137" watchObservedRunningTime="2026-07-24 16:07:43.632727311 +0000 UTC m=+83.064362672" Jul 24 16:07:43.647687 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.647640 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7gtd8" podStartSLOduration=72.274246817 podStartE2EDuration="1m22.647626345s" totalImagesPullingTime="10.373379528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:43.646770284 +0000 UTC m=+83.078405647" watchObservedRunningTime="2026-07-24 16:07:43.647626345 +0000 UTC m=+83.079261705" Jul 24 16:07:43.665464 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.665424 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-r2qp6" podStartSLOduration=12.661826683 podStartE2EDuration="22.66541215s" totalImagesPullingTime="10.003585467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:43.665348145 +0000 UTC m=+83.096983509" watchObservedRunningTime="2026-07-24 16:07:43.66541215 +0000 UTC m=+83.097047504" Jul 24 16:07:43.736934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.736908 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:43.806589 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:43.806561 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:45.508921 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.508890 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:45.689244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.689209 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:45.793692 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.793616 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm"] Jul 24 16:07:45.803434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.803410 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:45.863673 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.863638 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-szpx2"] Jul 24 16:07:45.863846 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.863764 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:45.870961 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.870940 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 24 16:07:45.871236 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.871220 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 24 16:07:45.871763 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.871746 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xtgt8\"" Jul 24 16:07:45.883375 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.883353 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f"] Jul 24 16:07:45.883486 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.883471 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.886400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.886378 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q9pb8\"" Jul 24 16:07:45.886400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.886386 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 16:07:45.886533 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.886382 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 16:07:45.886707 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.886692 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 16:07:45.902706 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.902689 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm"] Jul 24 16:07:45.902805 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.902725 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f"] Jul 24 16:07:45.902846 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.902809 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:45.906558 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.906533 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 24 16:07:45.906750 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.906736 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 24 16:07:45.906953 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.906938 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 24 16:07:45.907391 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.907376 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nql2w\"" Jul 24 16:07:45.926227 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926208 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-tls\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.926337 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926244 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abbc9b98-206f-41d7-b91a-cc701b707d4e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:45.926337 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926266 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:45.926337 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926281 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da368237-4ba5-4ddd-905c-c61f0ca37d08-root\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.926337 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926298 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-textfile\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.926474 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926352 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.926474 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926382 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abbc9b98-206f-41d7-b91a-cc701b707d4e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:45.926474 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926408 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h7rq\" (UniqueName: \"kubernetes.io/projected/da368237-4ba5-4ddd-905c-c61f0ca37d08-kube-api-access-2h7rq\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.926474 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926423 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-wtmp\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.926474 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926438 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da368237-4ba5-4ddd-905c-c61f0ca37d08-metrics-client-ca\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.926627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926486 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lk5k\" (UniqueName: \"kubernetes.io/projected/abbc9b98-206f-41d7-b91a-cc701b707d4e-kube-api-access-6lk5k\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:45.926627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926510 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3421be4c-1658-4f37-b7b4-363b802d0c69-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:45.926627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926539 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/abbc9b98-206f-41d7-b91a-cc701b707d4e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:45.926627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926558 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:45.926627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926575 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-accelerators-collector-config\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.926627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926589 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3421be4c-1658-4f37-b7b4-363b802d0c69-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:45.926800 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926631 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:45.926800 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926652 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h98w\" (UniqueName: \"kubernetes.io/projected/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-api-access-6h98w\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:45.926800 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.926666 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da368237-4ba5-4ddd-905c-c61f0ca37d08-sys\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:45.944648 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:45.944622 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:46.027675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027637 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/abbc9b98-206f-41d7-b91a-cc701b707d4e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.027675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.027915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027696 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-accelerators-collector-config\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.027915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027713 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3421be4c-1658-4f37-b7b4-363b802d0c69-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.027915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027747 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.027915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027769 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h98w\" (UniqueName: \"kubernetes.io/projected/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-api-access-6h98w\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.027915 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:46.027774 2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 24 16:07:46.027915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027784 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da368237-4ba5-4ddd-905c-c61f0ca37d08-sys\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.027915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027804 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-tls\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.027915 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:46.027845 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbc9b98-206f-41d7-b91a-cc701b707d4e-openshift-state-metrics-tls podName:abbc9b98-206f-41d7-b91a-cc701b707d4e nodeName:}" failed. No retries permitted until 2026-07-24 16:07:46.52782204 +0000 UTC m=+85.959457378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/abbc9b98-206f-41d7-b91a-cc701b707d4e-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-9qvsm" (UID: "abbc9b98-206f-41d7-b91a-cc701b707d4e") : secret "openshift-state-metrics-tls" not found Jul 24 16:07:46.027915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027891 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da368237-4ba5-4ddd-905c-c61f0ca37d08-sys\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027926 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abbc9b98-206f-41d7-b91a-cc701b707d4e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.027982 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028013 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da368237-4ba5-4ddd-905c-c61f0ca37d08-root\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-textfile\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028105 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da368237-4ba5-4ddd-905c-c61f0ca37d08-root\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028107 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028171 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abbc9b98-206f-41d7-b91a-cc701b707d4e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028220 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h7rq\" (UniqueName: \"kubernetes.io/projected/da368237-4ba5-4ddd-905c-c61f0ca37d08-kube-api-access-2h7rq\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028250 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-wtmp\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028276 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da368237-4ba5-4ddd-905c-c61f0ca37d08-metrics-client-ca\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028321 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lk5k\" (UniqueName: \"kubernetes.io/projected/abbc9b98-206f-41d7-b91a-cc701b707d4e-kube-api-access-6lk5k\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028346 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3421be4c-1658-4f37-b7b4-363b802d0c69-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.028406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028358 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-accelerators-collector-config\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.029000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028481 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-wtmp\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.029000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028765 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3421be4c-1658-4f37-b7b4-363b802d0c69-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.029000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.028905 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-textfile\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.029233 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.029213 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abbc9b98-206f-41d7-b91a-cc701b707d4e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.029314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.029266 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da368237-4ba5-4ddd-905c-c61f0ca37d08-metrics-client-ca\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.029407 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.029386 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.029755 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.029695 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3421be4c-1658-4f37-b7b4-363b802d0c69-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.031098 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.031065 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.031207 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.031074 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abbc9b98-206f-41d7-b91a-cc701b707d4e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.031207 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.031126 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.031287 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.031219 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.031364 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.031347 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da368237-4ba5-4ddd-905c-c61f0ca37d08-node-exporter-tls\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.041531 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.041508 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lk5k\" (UniqueName: \"kubernetes.io/projected/abbc9b98-206f-41d7-b91a-cc701b707d4e-kube-api-access-6lk5k\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.042407 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.042384 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h98w\" (UniqueName: \"kubernetes.io/projected/3421be4c-1658-4f37-b7b4-363b802d0c69-kube-api-access-6h98w\") pod \"kube-state-metrics-56d9f944b4-5nw7f\" (UID: \"3421be4c-1658-4f37-b7b4-363b802d0c69\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.056293 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.056246 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h7rq\" (UniqueName: \"kubernetes.io/projected/da368237-4ba5-4ddd-905c-c61f0ca37d08-kube-api-access-2h7rq\") pod \"node-exporter-szpx2\" (UID: \"da368237-4ba5-4ddd-905c-c61f0ca37d08\") " pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.127416 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.127382 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:46.191798 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.191772 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-szpx2" Jul 24 16:07:46.202258 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.202231 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:46.211229 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.211210 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" Jul 24 16:07:46.228542 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:46.228510 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda368237_4ba5_4ddd_905c_c61f0ca37d08.slice/crio-10bb5b5ce9ee2f008959e1606bb041ba03a7a9ea92668e770d596d3e43d1f128 WatchSource:0}: Error finding container 10bb5b5ce9ee2f008959e1606bb041ba03a7a9ea92668e770d596d3e43d1f128: Status 404 returned error can't find the container with id 10bb5b5ce9ee2f008959e1606bb041ba03a7a9ea92668e770d596d3e43d1f128 Jul 24 16:07:46.349442 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.349377 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:46.354969 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.354948 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f"] Jul 24 16:07:46.357269 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:46.357244 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3421be4c_1658_4f37_b7b4_363b802d0c69.slice/crio-5d0284a845389d34eaa012d3e23199ff7116c3189798b153361d968f535fb76e WatchSource:0}: Error finding container 5d0284a845389d34eaa012d3e23199ff7116c3189798b153361d968f535fb76e: Status 404 returned error can't find the container with id 5d0284a845389d34eaa012d3e23199ff7116c3189798b153361d968f535fb76e Jul 24 16:07:46.467012 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.466983 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:46.533427 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.533399 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/abbc9b98-206f-41d7-b91a-cc701b707d4e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.535835 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.535814 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/abbc9b98-206f-41d7-b91a-cc701b707d4e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9qvsm\" (UID: \"abbc9b98-206f-41d7-b91a-cc701b707d4e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.597312 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.597283 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" event={"ID":"3421be4c-1658-4f37-b7b4-363b802d0c69","Type":"ContainerStarted","Data":"5d0284a845389d34eaa012d3e23199ff7116c3189798b153361d968f535fb76e"} Jul 24 16:07:46.598277 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.598251 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-szpx2" event={"ID":"da368237-4ba5-4ddd-905c-c61f0ca37d08","Type":"ContainerStarted","Data":"10bb5b5ce9ee2f008959e1606bb041ba03a7a9ea92668e770d596d3e43d1f128"} Jul 24 16:07:46.772343 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.772313 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" Jul 24 16:07:46.813299 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.813264 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:46.910799 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.910772 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:46.922380 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.922351 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 16:07:46.947796 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.947770 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:46.952501 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.952210 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 16:07:46.952501 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.952368 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 16:07:46.952501 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.952377 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 16:07:46.952501 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.952468 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 16:07:46.952686 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.952634 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 16:07:46.952724 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.952710 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 16:07:46.952840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.952826 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 16:07:46.952899 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.952885 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pszxv\"" Jul 24 16:07:46.952977 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.952967 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 16:07:46.954285 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.954261 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm"] Jul 24 16:07:46.962501 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.962475 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 16:07:46.962782 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:46.962758 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 16:07:47.039273 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039247 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-out\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039343 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039303 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039384 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039343 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039429 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039406 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039449 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-web-config\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039519 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039557 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039523 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039557 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039547 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv8x7\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-kube-api-access-sv8x7\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039645 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039574 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039645 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039589 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039714 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039689 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039755 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039735 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.039835 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.039816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-volume\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.140824 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-web-config\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.140871 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.140906 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.140930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv8x7\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-kube-api-access-sv8x7\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.140952 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.140991 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.141015 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.141161 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.141238 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-volume\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.141275 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-out\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.141317 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.141344 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.141565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.141369 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.142169 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.142085 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.142982 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.142396 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.143677 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.143551 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.145625 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.144942 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.145625 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.145414 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-out\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.145625 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.145578 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-web-config\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.145840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.145719 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.146880 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.146031 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.147658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.147595 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.147901 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.147878 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.148061 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.148025 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.148356 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.148128 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-volume\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.150276 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.150255 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv8x7\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-kube-api-access-sv8x7\") pod \"alertmanager-main-0\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.235082 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.235055 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:47.264158 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.264109 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:07:47.533943 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.533919 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 16:07:47.534821 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:47.534796 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb789ec5a_e174_4c49_8c9d_26d7b5865c94.slice/crio-1644695a0276e5370e133b1d439b6f5042cecc26cf5bb723fdb4155d916d1eae WatchSource:0}: Error finding container 1644695a0276e5370e133b1d439b6f5042cecc26cf5bb723fdb4155d916d1eae: Status 404 returned error can't find the container with id 1644695a0276e5370e133b1d439b6f5042cecc26cf5bb723fdb4155d916d1eae Jul 24 16:07:47.603895 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.603865 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" event={"ID":"abbc9b98-206f-41d7-b91a-cc701b707d4e","Type":"ContainerStarted","Data":"99af6f582ad24120e6cc3af663874181b598cf8f33b4f3d38d67ead2285a9cff"} Jul 24 16:07:47.604054 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.603930 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" event={"ID":"abbc9b98-206f-41d7-b91a-cc701b707d4e","Type":"ContainerStarted","Data":"802ca10b0a52e6efb112535a5afe2f18321c6e973cb5e74da48e9f8ba3f2efcc"} Jul 24 16:07:47.604054 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.603958 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" event={"ID":"abbc9b98-206f-41d7-b91a-cc701b707d4e","Type":"ContainerStarted","Data":"21fc923c0cf37b70b7d9e265ebf46410e38f9384ec29d5066bb0390adbf2aed1"} Jul 24 16:07:47.605418 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:47.605394 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerStarted","Data":"1644695a0276e5370e133b1d439b6f5042cecc26cf5bb723fdb4155d916d1eae"} Jul 24 16:07:48.612787 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.612750 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" event={"ID":"3421be4c-1658-4f37-b7b4-363b802d0c69","Type":"ContainerStarted","Data":"3a38f97d66a790b6aab6ee78a0da2d8f2eea2e16183e0d17ece3688f2ebcffef"} Jul 24 16:07:48.613230 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.612859 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" event={"ID":"3421be4c-1658-4f37-b7b4-363b802d0c69","Type":"ContainerStarted","Data":"65ec751b12c195a6f12ab108ce77aa848b7f376f99cff1d23e932b8a9d0b2980"} Jul 24 16:07:48.616359 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.616326 2560 generic.go:356] "Generic (PLEG): container finished" podID="da368237-4ba5-4ddd-905c-c61f0ca37d08" containerID="2cd04bc50ed6d48025a1cbfb4f60d21652f71eadb9e67231913f16d6e51c91f8" exitCode=0 Jul 24 16:07:48.616559 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.616375 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-szpx2" event={"ID":"da368237-4ba5-4ddd-905c-c61f0ca37d08","Type":"ContainerDied","Data":"2cd04bc50ed6d48025a1cbfb4f60d21652f71eadb9e67231913f16d6e51c91f8"} Jul 24 16:07:48.766227 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.766102 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:48.850137 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.850110 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-769c878fbf-9zdq6"] Jul 24 16:07:48.875336 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.875194 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:48.878919 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.878806 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-769c878fbf-9zdq6"] Jul 24 16:07:48.879957 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.879508 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 24 16:07:48.879957 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.879931 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-84496\"" Jul 24 16:07:48.880331 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.880136 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 24 16:07:48.880331 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.880177 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 24 16:07:48.880533 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.880420 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 24 16:07:48.880784 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.880602 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 24 16:07:48.880885 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.880817 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-19aeelgn9u01c\"" Jul 24 16:07:48.899352 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.899317 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:48.958045 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.958013 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thntw\" (UniqueName: \"kubernetes.io/projected/f319b5fc-461d-4665-a37a-5317b77e9bf9-kube-api-access-thntw\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:48.958252 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.958063 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:48.958252 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.958130 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:48.958252 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.958190 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-tls\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:48.958252 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.958227 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f319b5fc-461d-4665-a37a-5317b77e9bf9-metrics-client-ca\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:48.958252 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.958241 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-grpc-tls\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:48.958446 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.958259 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:48.958446 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.958348 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:48.999915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:48.999873 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:49.059081 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.059047 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.059260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.059097 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thntw\" (UniqueName: \"kubernetes.io/projected/f319b5fc-461d-4665-a37a-5317b77e9bf9-kube-api-access-thntw\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.059260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.059136 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.059260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.059199 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.059260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.059238 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-tls\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.059475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.059288 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f319b5fc-461d-4665-a37a-5317b77e9bf9-metrics-client-ca\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.059475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.059316 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-grpc-tls\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.059475 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.059344 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.060358 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.060323 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f319b5fc-461d-4665-a37a-5317b77e9bf9-metrics-client-ca\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.062172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.062127 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.062393 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.062374 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.062570 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.062537 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-grpc-tls\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.062570 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.062548 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-tls\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.062739 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.062677 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.062873 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.062854 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f319b5fc-461d-4665-a37a-5317b77e9bf9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.068444 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.068423 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thntw\" (UniqueName: \"kubernetes.io/projected/f319b5fc-461d-4665-a37a-5317b77e9bf9-kube-api-access-thntw\") pod \"thanos-querier-769c878fbf-9zdq6\" (UID: \"f319b5fc-461d-4665-a37a-5317b77e9bf9\") " pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.194975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.194903 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:49.623686 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.622788 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" event={"ID":"3421be4c-1658-4f37-b7b4-363b802d0c69","Type":"ContainerStarted","Data":"de45a9803b772cf662c57743168a1eb437dd56e622d25d927036f8d54ce3662c"} Jul 24 16:07:49.626517 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.626493 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-szpx2" event={"ID":"da368237-4ba5-4ddd-905c-c61f0ca37d08","Type":"ContainerStarted","Data":"86c33a08e07686448b57473338603993fbc6a3d4b926589c7445412e6fc6568c"} Jul 24 16:07:49.626630 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.626536 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-szpx2" event={"ID":"da368237-4ba5-4ddd-905c-c61f0ca37d08","Type":"ContainerStarted","Data":"f39702dba0b861ee99d9e4b244cecc73dfd0d624512b7f9c2fba3320e6974d17"} Jul 24 16:07:49.645679 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.645634 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5nw7f" podStartSLOduration=2.642818429 podStartE2EDuration="4.645620503s" totalImagesPullingTime="2.002802074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:49.644566192 +0000 UTC m=+89.076201572" watchObservedRunningTime="2026-07-24 16:07:49.645620503 +0000 UTC m=+89.077255864" Jul 24 16:07:49.665890 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.665844 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-szpx2" podStartSLOduration=2.7324221079999997 podStartE2EDuration="4.665828735s" totalImagesPullingTime="1.855351449s" totalInitContainerRuntime="78.055178ms" isStatefulPod=true podCreationTimestamp="2026-07-24 16:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:49.665493091 +0000 UTC m=+89.097128451" watchObservedRunningTime="2026-07-24 16:07:49.665828735 +0000 UTC m=+89.097464141" Jul 24 16:07:49.957780 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.957758 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-769c878fbf-9zdq6"] Jul 24 16:07:49.959677 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:49.959652 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf319b5fc_461d_4665_a37a_5317b77e9bf9.slice/crio-cdcf06d722d7430da9f1746ad76d80135a6efb18b4b0a3523e8e72be88e49ca5 WatchSource:0}: Error finding container cdcf06d722d7430da9f1746ad76d80135a6efb18b4b0a3523e8e72be88e49ca5: Status 404 returned error can't find the container with id cdcf06d722d7430da9f1746ad76d80135a6efb18b4b0a3523e8e72be88e49ca5 Jul 24 16:07:49.962627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:49.962594 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:50.035922 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.035866 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:50.187761 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.187692 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f9844cddc-ksj4r"] Jul 24 16:07:50.212977 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.212946 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f9844cddc-ksj4r"] Jul 24 16:07:50.213135 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.213081 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.215693 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.215669 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 24 16:07:50.215822 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.215732 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ehaff6fc6cl2g\"" Jul 24 16:07:50.215822 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.215670 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 16:07:50.216006 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.215988 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 24 16:07:50.216176 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.216083 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 24 16:07:50.216612 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.216597 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qmsld\"" Jul 24 16:07:50.268856 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.268837 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-secret-metrics-server-client-certs\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.268856 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.268863 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-client-ca-bundle\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.269055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.268886 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-metrics-server-audit-profiles\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.269055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.268906 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-audit-log\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.269055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.268927 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.269055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.269005 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-secret-metrics-server-tls\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.269055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.269037 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwkr\" (UniqueName: \"kubernetes.io/projected/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-kube-api-access-8zwkr\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.369839 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.369810 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.369947 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.369867 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-secret-metrics-server-tls\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.369947 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.369902 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwkr\" (UniqueName: \"kubernetes.io/projected/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-kube-api-access-8zwkr\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.370061 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.370041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-secret-metrics-server-client-certs\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.370116 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.370078 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-client-ca-bundle\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.370116 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.370104 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-metrics-server-audit-profiles\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.370240 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.370136 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-audit-log\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.370552 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.370530 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-audit-log\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.370805 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.370564 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.371009 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.370985 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-metrics-server-audit-profiles\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.372379 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.372357 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-secret-metrics-server-tls\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.372465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.372396 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-secret-metrics-server-client-certs\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.372465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.372426 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-client-ca-bundle\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.380209 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.380172 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwkr\" (UniqueName: \"kubernetes.io/projected/d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e-kube-api-access-8zwkr\") pod \"metrics-server-f9844cddc-ksj4r\" (UID: \"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e\") " pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.522654 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.522588 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:07:50.635818 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.634910 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" event={"ID":"abbc9b98-206f-41d7-b91a-cc701b707d4e","Type":"ContainerStarted","Data":"1fb999b4358dc60a2536628726090000071d7ad422a7d5e3f2d54515e489b819"} Jul 24 16:07:50.637599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.637569 2560 generic.go:356] "Generic (PLEG): container finished" podID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerID="e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e" exitCode=0 Jul 24 16:07:50.637729 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.637670 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerDied","Data":"e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e"} Jul 24 16:07:50.639841 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.639816 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" event={"ID":"f319b5fc-461d-4665-a37a-5317b77e9bf9","Type":"ContainerStarted","Data":"cdcf06d722d7430da9f1746ad76d80135a6efb18b4b0a3523e8e72be88e49ca5"} Jul 24 16:07:50.649305 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.649279 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77d8b4774d-qrsl2"] Jul 24 16:07:50.667689 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.667573 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.673352 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.673324 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9"] Jul 24 16:07:50.675043 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.674986 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:50.687765 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.687724 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d8b4774d-qrsl2"] Jul 24 16:07:50.687866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.687837 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:50.692112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.691133 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9qvsm" podStartSLOduration=3.367126343 podStartE2EDuration="5.691113405s" totalImagesPullingTime="2.323987062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:50.684766607 +0000 UTC m=+90.116401968" watchObservedRunningTime="2026-07-24 16:07:50.691113405 +0000 UTC m=+90.122748766" Jul 24 16:07:50.693157 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.693119 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-54hbd\"" Jul 24 16:07:50.693570 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.693555 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9"] Jul 24 16:07:50.695174 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.695133 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f9844cddc-ksj4r"] Jul 24 16:07:50.695835 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.695797 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 24 16:07:50.700935 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:50.700912 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75a1be4_84f6_44b9_95a0_55e2d7d5ea3e.slice/crio-a9ed7fdaf977fe4bc39b89087645f2ebdfee702a9fab9db0aac1f412d266dad6 WatchSource:0}: Error finding container a9ed7fdaf977fe4bc39b89087645f2ebdfee702a9fab9db0aac1f412d266dad6: Status 404 returned error can't find the container with id a9ed7fdaf977fe4bc39b89087645f2ebdfee702a9fab9db0aac1f412d266dad6 Jul 24 16:07:50.765504 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.765461 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:50.774323 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.774106 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mbd\" (UniqueName: \"kubernetes.io/projected/a8b4e365-3ff3-45f8-8914-9236e5aa033d-kube-api-access-54mbd\") pod \"monitoring-plugin-68b7fc9574-7tnn9\" (UID: \"a8b4e365-3ff3-45f8-8914-9236e5aa033d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:50.774323 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.774197 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-trusted-ca-bundle\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.774323 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.774230 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-service-ca\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.774323 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.774255 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-oauth-config\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.774323 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.774314 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8b4e365-3ff3-45f8-8914-9236e5aa033d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7tnn9\" (UID: \"a8b4e365-3ff3-45f8-8914-9236e5aa033d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:50.774581 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.774356 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-oauth-serving-cert\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.774581 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.774387 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-config\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.774581 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.774418 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-serving-cert\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.774581 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.774487 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7877d\" (UniqueName: \"kubernetes.io/projected/26ef84d1-6d96-4866-9eb3-72cebcb287f8-kube-api-access-7877d\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.829569 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.829535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:50.875824 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.875797 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-trusted-ca-bundle\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.875988 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.875832 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-service-ca\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.875988 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.875855 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-oauth-config\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.875988 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.875874 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8b4e365-3ff3-45f8-8914-9236e5aa033d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7tnn9\" (UID: \"a8b4e365-3ff3-45f8-8914-9236e5aa033d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:50.875988 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:50.875957 2560 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 24 16:07:50.876224 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:07:50.876008 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b4e365-3ff3-45f8-8914-9236e5aa033d-monitoring-plugin-cert podName:a8b4e365-3ff3-45f8-8914-9236e5aa033d nodeName:}" failed. No retries permitted until 2026-07-24 16:07:51.37599257 +0000 UTC m=+90.807627910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a8b4e365-3ff3-45f8-8914-9236e5aa033d-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-7tnn9" (UID: "a8b4e365-3ff3-45f8-8914-9236e5aa033d") : secret "monitoring-plugin-cert" not found Jul 24 16:07:50.876224 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.876001 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-oauth-serving-cert\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.876224 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.876056 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-config\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.876224 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.876091 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-serving-cert\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.876224 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.876139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7877d\" (UniqueName: \"kubernetes.io/projected/26ef84d1-6d96-4866-9eb3-72cebcb287f8-kube-api-access-7877d\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.876470 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.876247 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mbd\" (UniqueName: \"kubernetes.io/projected/a8b4e365-3ff3-45f8-8914-9236e5aa033d-kube-api-access-54mbd\") pod \"monitoring-plugin-68b7fc9574-7tnn9\" (UID: \"a8b4e365-3ff3-45f8-8914-9236e5aa033d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:50.876852 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.876827 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-service-ca\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.876949 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.876879 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-trusted-ca-bundle\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.877062 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.877042 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-config\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.877327 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.877305 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-oauth-serving-cert\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.878513 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.878490 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-serving-cert\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.878590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.878528 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-oauth-config\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.884374 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.884344 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mbd\" (UniqueName: \"kubernetes.io/projected/a8b4e365-3ff3-45f8-8914-9236e5aa033d-kube-api-access-54mbd\") pod \"monitoring-plugin-68b7fc9574-7tnn9\" (UID: \"a8b4e365-3ff3-45f8-8914-9236e5aa033d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:50.884909 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.884895 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7877d\" (UniqueName: \"kubernetes.io/projected/26ef84d1-6d96-4866-9eb3-72cebcb287f8-kube-api-access-7877d\") pod \"console-77d8b4774d-qrsl2\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:50.980308 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:50.980273 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:07:51.139084 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:51.139054 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:51.158740 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:51.158710 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d8b4774d-qrsl2"] Jul 24 16:07:51.164479 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:51.164449 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ef84d1_6d96_4866_9eb3_72cebcb287f8.slice/crio-1b10200cb22aa92e78a2d016b36c42fd654c4ecf96cf441a9b591573ed6594ce WatchSource:0}: Error finding container 1b10200cb22aa92e78a2d016b36c42fd654c4ecf96cf441a9b591573ed6594ce: Status 404 returned error can't find the container with id 1b10200cb22aa92e78a2d016b36c42fd654c4ecf96cf441a9b591573ed6594ce Jul 24 16:07:51.381445 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:51.381405 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8b4e365-3ff3-45f8-8914-9236e5aa033d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7tnn9\" (UID: \"a8b4e365-3ff3-45f8-8914-9236e5aa033d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:51.386168 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:51.386120 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8b4e365-3ff3-45f8-8914-9236e5aa033d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7tnn9\" (UID: \"a8b4e365-3ff3-45f8-8914-9236e5aa033d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:51.601248 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:51.601211 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:51.648244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:51.648172 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d8b4774d-qrsl2" event={"ID":"26ef84d1-6d96-4866-9eb3-72cebcb287f8","Type":"ContainerStarted","Data":"e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809"} Jul 24 16:07:51.648676 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:51.648265 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d8b4774d-qrsl2" event={"ID":"26ef84d1-6d96-4866-9eb3-72cebcb287f8","Type":"ContainerStarted","Data":"1b10200cb22aa92e78a2d016b36c42fd654c4ecf96cf441a9b591573ed6594ce"} Jul 24 16:07:51.650202 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:51.650175 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" event={"ID":"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e","Type":"ContainerStarted","Data":"a9ed7fdaf977fe4bc39b89087645f2ebdfee702a9fab9db0aac1f412d266dad6"} Jul 24 16:07:51.672201 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:51.672138 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77d8b4774d-qrsl2" podStartSLOduration=1.672117638 podStartE2EDuration="1.672117638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:51.67001102 +0000 UTC m=+91.101646381" watchObservedRunningTime="2026-07-24 16:07:51.672117638 +0000 UTC m=+91.103752999" Jul 24 16:07:52.162169 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.162085 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:52.163559 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.163536 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 16:07:52.178501 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.178483 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.184205 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.184120 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 16:07:52.184333 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.184210 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 16:07:52.184333 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.184216 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 16:07:52.184333 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.184223 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 16:07:52.184333 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.184233 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 16:07:52.184518 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.184498 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5gq2v\"" Jul 24 16:07:52.184584 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.184526 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-albv6s7ruqn2n\"" Jul 24 16:07:52.184584 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.184559 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 16:07:52.185056 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.185040 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 16:07:52.185385 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.185366 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 16:07:52.185484 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.185434 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 16:07:52.185646 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.185623 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 16:07:52.185933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.185907 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 16:07:52.188115 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.187518 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 16:07:52.191779 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.191716 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 16:07:52.194721 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.194635 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 16:07:52.196137 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.196115 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 16:07:52.292666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.292634 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.292834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.292681 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.292834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.292759 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.292834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.292798 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.292834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.292826 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293059 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.292900 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg5c2\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-kube-api-access-bg5c2\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293059 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.292930 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293059 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.292964 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293059 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293014 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293059 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293036 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293365 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293062 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293365 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293106 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293365 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293170 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293365 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293207 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293365 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293241 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293365 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293299 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293365 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293327 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293365 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293358 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293751 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293376 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.293751 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.293401 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396237 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396193 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396255 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396287 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396328 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396368 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396392 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396420 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396479 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396522 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396556 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396582 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396612 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396637 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.396666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396667 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg5c2\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-kube-api-access-bg5c2\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.397094 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.397094 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396730 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.397094 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396756 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.397094 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396774 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.397094 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.396789 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.397383 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.397204 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.397383 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.397241 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.397878 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.397537 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.397998 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.397912 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.398449 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.398425 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.398965 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.398940 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.401055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.400628 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.401055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.400712 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.401055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.400944 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.401055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.401019 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.401339 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.401296 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.402881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.402854 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.403812 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.403774 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.403812 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.403789 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.403812 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.403799 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.404003 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.403969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.407103 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.404210 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.407103 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.404705 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.407103 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.405654 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.409927 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.409909 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg5c2\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-kube-api-access-bg5c2\") pod \"prometheus-k8s-0\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:52.490808 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:52.490721 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:07:53.771649 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:53.771583 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:53.823981 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:53.823945 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9"] Jul 24 16:07:53.827954 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:53.827932 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b4e365_3ff3_45f8_8914_9236e5aa033d.slice/crio-fb6f19050ce09675d115d29aa19242ee797ef9abd85338e78a8821f1c04c61c2 WatchSource:0}: Error finding container fb6f19050ce09675d115d29aa19242ee797ef9abd85338e78a8821f1c04c61c2: Status 404 returned error can't find the container with id fb6f19050ce09675d115d29aa19242ee797ef9abd85338e78a8821f1c04c61c2 Jul 24 16:07:53.865789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:53.865690 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 16:07:53.868572 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:07:53.868544 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a083fae_642e_4b16_acf9_f9c5d721f5cc.slice/crio-7b9f3fd7ba54bddb438933f0c679f0c9808d4971b19db3896155b0db699670f7 WatchSource:0}: Error finding container 7b9f3fd7ba54bddb438933f0c679f0c9808d4971b19db3896155b0db699670f7: Status 404 returned error can't find the container with id 7b9f3fd7ba54bddb438933f0c679f0c9808d4971b19db3896155b0db699670f7 Jul 24 16:07:53.872589 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:53.872529 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:53.942570 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:53.942504 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:07:54.666481 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.666447 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerID="1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9" exitCode=0 Jul 24 16:07:54.666634 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.666535 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerDied","Data":"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9"} Jul 24 16:07:54.666634 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.666621 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerStarted","Data":"7b9f3fd7ba54bddb438933f0c679f0c9808d4971b19db3896155b0db699670f7"} Jul 24 16:07:54.667961 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.667752 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" event={"ID":"a8b4e365-3ff3-45f8-8914-9236e5aa033d","Type":"ContainerStarted","Data":"fb6f19050ce09675d115d29aa19242ee797ef9abd85338e78a8821f1c04c61c2"} Jul 24 16:07:54.671191 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.671157 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerStarted","Data":"5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686"} Jul 24 16:07:54.671275 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.671236 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerStarted","Data":"06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b"} Jul 24 16:07:54.671330 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.671277 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerStarted","Data":"4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa"} Jul 24 16:07:54.671330 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.671318 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerStarted","Data":"7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090"} Jul 24 16:07:54.671434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.671356 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerStarted","Data":"fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d"} Jul 24 16:07:54.673370 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.673349 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" event={"ID":"f319b5fc-461d-4665-a37a-5317b77e9bf9","Type":"ContainerStarted","Data":"c44394c4c71f3c5bb5de64a465b331865d86c186bf175e5a4c9747596e3da2d4"} Jul 24 16:07:54.673453 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.673414 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" event={"ID":"f319b5fc-461d-4665-a37a-5317b77e9bf9","Type":"ContainerStarted","Data":"d0bd26b01662166bf4b65fcf7ff7ca16f949e255cc66b72022b8fa7ee1b6c013"} Jul 24 16:07:54.673502 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.673454 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" event={"ID":"f319b5fc-461d-4665-a37a-5317b77e9bf9","Type":"ContainerStarted","Data":"6535677ab71e91f6861bc10bb17b1e0536735517f49852f5af5fdfd2d5d1bcbc"} Jul 24 16:07:54.674879 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.674847 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" event={"ID":"d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e","Type":"ContainerStarted","Data":"02829bdd50078471c16ecc1955f3e133c6f768c9a3d11f0d2b8cb191b94a49d5"} Jul 24 16:07:54.730413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:54.730362 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" podStartSLOduration=1.77652687 podStartE2EDuration="4.730347821s" totalImagesPullingTime="2.953820951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:54.728272915 +0000 UTC m=+94.159908276" watchObservedRunningTime="2026-07-24 16:07:54.730347821 +0000 UTC m=+94.161983180" Jul 24 16:07:56.686908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.686595 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" event={"ID":"a8b4e365-3ff3-45f8-8914-9236e5aa033d","Type":"ContainerStarted","Data":"087a64790b1e5cca06495301d2f169f5e2f3656e4c30b312cc2e3e32fd3d3d31"} Jul 24 16:07:56.686908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.686658 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:56.691706 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.691673 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerStarted","Data":"3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb"} Jul 24 16:07:56.693547 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.693525 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" Jul 24 16:07:56.695178 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.695135 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" event={"ID":"f319b5fc-461d-4665-a37a-5317b77e9bf9","Type":"ContainerStarted","Data":"99c4975dda7e9d72a3ac9cebf6401c51d66fbf950787adf67fcc88ee5a9033a4"} Jul 24 16:07:56.695292 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.695216 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" event={"ID":"f319b5fc-461d-4665-a37a-5317b77e9bf9","Type":"ContainerStarted","Data":"b3f6d8d84b4cf0674b0b10b4bd46c8d2c07239737f0d9552fe4b9d6e792f6b8b"} Jul 24 16:07:56.695292 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.695257 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" event={"ID":"f319b5fc-461d-4665-a37a-5317b77e9bf9","Type":"ContainerStarted","Data":"a97ac2e1c0a0cb20b02ab3fe2eb8bfd0d4522c753628fd7ff599f9b6fcd36212"} Jul 24 16:07:56.695381 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.695313 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:07:56.703278 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.703232 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tnn9" podStartSLOduration=4.482961666 podStartE2EDuration="6.703212772s" totalImagesPullingTime="2.220251106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:56.703121372 +0000 UTC m=+96.134756737" watchObservedRunningTime="2026-07-24 16:07:56.703212772 +0000 UTC m=+96.134848135" Jul 24 16:07:56.726892 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.726840 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" podStartSLOduration=3.132235109 podStartE2EDuration="8.726819095s" totalImagesPullingTime="5.594583986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:56.72582106 +0000 UTC m=+96.157456646" watchObservedRunningTime="2026-07-24 16:07:56.726819095 +0000 UTC m=+96.158454455" Jul 24 16:07:56.784029 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:56.783977 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.780927359 podStartE2EDuration="10.783954773s" totalImagesPullingTime="6.985371915s" totalInitContainerRuntime="17.655499ms" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:56.761178096 +0000 UTC m=+96.192813457" watchObservedRunningTime="2026-07-24 16:07:56.783954773 +0000 UTC m=+96.215590135" Jul 24 16:07:59.712408 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:59.712369 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerStarted","Data":"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21"} Jul 24 16:07:59.712848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:59.712433 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerStarted","Data":"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6"} Jul 24 16:07:59.712848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:59.712460 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerStarted","Data":"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94"} Jul 24 16:07:59.712848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:59.712482 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerStarted","Data":"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2"} Jul 24 16:07:59.712848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:59.712506 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerStarted","Data":"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1"} Jul 24 16:07:59.712848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:59.712529 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerStarted","Data":"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646"} Jul 24 16:07:59.746805 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:07:59.746742 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.572634249 podStartE2EDuration="7.74672415s" totalImagesPullingTime="4.133311024s" totalInitContainerRuntime="40.778877ms" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:59.744713732 +0000 UTC m=+99.176349094" watchObservedRunningTime="2026-07-24 16:07:59.74672415 +0000 UTC m=+99.178359509" Jul 24 16:08:00.981079 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:00.981041 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:08:00.981559 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:00.981122 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:08:00.985768 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:00.985748 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:08:01.639193 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.639133 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fcb847f95-h2w7j" podUID="727735e9-6b62-45d0-adca-6b988fe783b9" containerName="console" containerID="cri-o://4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0" gracePeriod=15 Jul 24 16:08:01.724136 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.724108 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:08:01.777431 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.777403 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87f758789-jc6mj"] Jul 24 16:08:01.918299 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.918276 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:08:01.931137 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.931119 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:08:01.998698 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.998673 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-oauth-serving-cert\") pod \"727735e9-6b62-45d0-adca-6b988fe783b9\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " Jul 24 16:08:01.999076 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.998715 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsmcq\" (UniqueName: \"kubernetes.io/projected/727735e9-6b62-45d0-adca-6b988fe783b9-kube-api-access-fsmcq\") pod \"727735e9-6b62-45d0-adca-6b988fe783b9\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " Jul 24 16:08:01.999076 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.998739 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-oauth-config\") pod \"727735e9-6b62-45d0-adca-6b988fe783b9\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " Jul 24 16:08:01.999076 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.998763 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-service-ca\") pod \"727735e9-6b62-45d0-adca-6b988fe783b9\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " Jul 24 16:08:01.999076 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.998785 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-serving-cert\") pod \"727735e9-6b62-45d0-adca-6b988fe783b9\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " Jul 24 16:08:01.999076 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.998812 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-console-config\") pod \"727735e9-6b62-45d0-adca-6b988fe783b9\" (UID: \"727735e9-6b62-45d0-adca-6b988fe783b9\") " Jul 24 16:08:01.999377 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.999124 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "727735e9-6b62-45d0-adca-6b988fe783b9" (UID: "727735e9-6b62-45d0-adca-6b988fe783b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:08:01.999377 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.999167 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-service-ca" (OuterVolumeSpecName: "service-ca") pod "727735e9-6b62-45d0-adca-6b988fe783b9" (UID: "727735e9-6b62-45d0-adca-6b988fe783b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:08:01.999377 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:01.999342 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-console-config" (OuterVolumeSpecName: "console-config") pod "727735e9-6b62-45d0-adca-6b988fe783b9" (UID: "727735e9-6b62-45d0-adca-6b988fe783b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:08:02.001270 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.001240 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "727735e9-6b62-45d0-adca-6b988fe783b9" (UID: "727735e9-6b62-45d0-adca-6b988fe783b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:08:02.001391 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.001315 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727735e9-6b62-45d0-adca-6b988fe783b9-kube-api-access-fsmcq" (OuterVolumeSpecName: "kube-api-access-fsmcq") pod "727735e9-6b62-45d0-adca-6b988fe783b9" (UID: "727735e9-6b62-45d0-adca-6b988fe783b9"). InnerVolumeSpecName "kube-api-access-fsmcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:08:02.001391 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.001337 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "727735e9-6b62-45d0-adca-6b988fe783b9" (UID: "727735e9-6b62-45d0-adca-6b988fe783b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:08:02.017472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.017413 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:08:02.100021 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.099996 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-oauth-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:02.100021 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.100017 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsmcq\" (UniqueName: \"kubernetes.io/projected/727735e9-6b62-45d0-adca-6b988fe783b9-kube-api-access-fsmcq\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:02.100180 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.100027 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-oauth-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:02.100180 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.100036 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:02.100180 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.100044 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727735e9-6b62-45d0-adca-6b988fe783b9-console-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:02.100180 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.100052 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727735e9-6b62-45d0-adca-6b988fe783b9-console-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:02.118194 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.118132 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:08:02.181034 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.180964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:08:02.491303 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.491225 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:08:02.706955 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.706931 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-769c878fbf-9zdq6" Jul 24 16:08:02.725818 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.725783 2560 generic.go:356] "Generic (PLEG): container finished" podID="727735e9-6b62-45d0-adca-6b988fe783b9" containerID="4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0" exitCode=0 Jul 24 16:08:02.725934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.725835 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fcb847f95-h2w7j" Jul 24 16:08:02.725934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.725864 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fcb847f95-h2w7j" event={"ID":"727735e9-6b62-45d0-adca-6b988fe783b9","Type":"ContainerDied","Data":"4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0"} Jul 24 16:08:02.725934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.725917 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fcb847f95-h2w7j" event={"ID":"727735e9-6b62-45d0-adca-6b988fe783b9","Type":"ContainerDied","Data":"370fcaff425af3e80ce16cf1cac2236d2ed4c289964e9b8ba36966a9b22c80f8"} Jul 24 16:08:02.726053 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.725937 2560 scope.go:117] "RemoveContainer" containerID="4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0" Jul 24 16:08:02.735016 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.734998 2560 scope.go:117] "RemoveContainer" containerID="4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0" Jul 24 16:08:02.735357 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:08:02.735335 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0\": container with ID starting with 4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0 not found: ID does not exist" containerID="4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0" Jul 24 16:08:02.735414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.735369 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0"} err="failed to get container status \"4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0\": rpc error: code = NotFound desc = could not find container \"4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0\": container with ID starting with 4c87704ab78746573af5068e2ab84d779f7ae3f4238caa50779ab0a17e338fc0 not found: ID does not exist" Jul 24 16:08:02.755904 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.755818 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fcb847f95-h2w7j"] Jul 24 16:08:02.757992 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:02.757972 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fcb847f95-h2w7j"] Jul 24 16:08:03.165536 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:03.165501 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727735e9-6b62-45d0-adca-6b988fe783b9" path="/var/lib/kubelet/pods/727735e9-6b62-45d0-adca-6b988fe783b9/volumes" Jul 24 16:08:04.940014 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:04.939982 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:08:05.005613 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:05.005565 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:08:10.523639 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:10.523605 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:08:10.524089 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:10.523662 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:08:14.594158 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:14.594102 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-42m7w" Jul 24 16:08:26.797893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:26.797823 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-87f758789-jc6mj" podUID="61cbab77-e0a5-48d3-97bc-7e3af545a45b" containerName="console" containerID="cri-o://60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf" gracePeriod=15 Jul 24 16:08:27.571567 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.571545 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:08:27.626003 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.625972 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-config\") pod \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " Jul 24 16:08:27.626165 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626022 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-trusted-ca-bundle\") pod \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " Jul 24 16:08:27.626165 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626060 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-service-ca\") pod \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " Jul 24 16:08:27.626165 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626080 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-serving-cert\") pod \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " Jul 24 16:08:27.626165 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626104 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kfx4\" (UniqueName: \"kubernetes.io/projected/61cbab77-e0a5-48d3-97bc-7e3af545a45b-kube-api-access-5kfx4\") pod \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " Jul 24 16:08:27.626165 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626130 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-oauth-config\") pod \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " Jul 24 16:08:27.626423 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626184 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-oauth-serving-cert\") pod \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\" (UID: \"61cbab77-e0a5-48d3-97bc-7e3af545a45b\") " Jul 24 16:08:27.626478 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626411 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-config" (OuterVolumeSpecName: "console-config") pod "61cbab77-e0a5-48d3-97bc-7e3af545a45b" (UID: "61cbab77-e0a5-48d3-97bc-7e3af545a45b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:08:27.626478 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626460 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "61cbab77-e0a5-48d3-97bc-7e3af545a45b" (UID: "61cbab77-e0a5-48d3-97bc-7e3af545a45b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:08:27.626587 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626494 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-service-ca" (OuterVolumeSpecName: "service-ca") pod "61cbab77-e0a5-48d3-97bc-7e3af545a45b" (UID: "61cbab77-e0a5-48d3-97bc-7e3af545a45b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:08:27.626720 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.626701 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "61cbab77-e0a5-48d3-97bc-7e3af545a45b" (UID: "61cbab77-e0a5-48d3-97bc-7e3af545a45b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:08:27.628486 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.628448 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "61cbab77-e0a5-48d3-97bc-7e3af545a45b" (UID: "61cbab77-e0a5-48d3-97bc-7e3af545a45b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:08:27.628599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.628545 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "61cbab77-e0a5-48d3-97bc-7e3af545a45b" (UID: "61cbab77-e0a5-48d3-97bc-7e3af545a45b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:08:27.628599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.628554 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cbab77-e0a5-48d3-97bc-7e3af545a45b-kube-api-access-5kfx4" (OuterVolumeSpecName: "kube-api-access-5kfx4") pod "61cbab77-e0a5-48d3-97bc-7e3af545a45b" (UID: "61cbab77-e0a5-48d3-97bc-7e3af545a45b"). InnerVolumeSpecName "kube-api-access-5kfx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:08:27.727255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.727195 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:27.727255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.727216 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-trusted-ca-bundle\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:27.727255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.727225 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:27.727255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.727234 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:27.727255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.727243 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kfx4\" (UniqueName: \"kubernetes.io/projected/61cbab77-e0a5-48d3-97bc-7e3af545a45b-kube-api-access-5kfx4\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:27.727255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.727252 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61cbab77-e0a5-48d3-97bc-7e3af545a45b-console-oauth-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:27.727520 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.727260 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61cbab77-e0a5-48d3-97bc-7e3af545a45b-oauth-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:08:27.833874 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.833840 2560 generic.go:356] "Generic (PLEG): container finished" podID="61cbab77-e0a5-48d3-97bc-7e3af545a45b" containerID="60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf" exitCode=0 Jul 24 16:08:27.834234 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.833886 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87f758789-jc6mj" Jul 24 16:08:27.834234 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.833922 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87f758789-jc6mj" event={"ID":"61cbab77-e0a5-48d3-97bc-7e3af545a45b","Type":"ContainerDied","Data":"60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf"} Jul 24 16:08:27.834234 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.833973 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87f758789-jc6mj" event={"ID":"61cbab77-e0a5-48d3-97bc-7e3af545a45b","Type":"ContainerDied","Data":"cbdf57abe4715f87bf8333a87011d5df91de6d7c973f5594e7a9f768d591514d"} Jul 24 16:08:27.834234 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.833993 2560 scope.go:117] "RemoveContainer" containerID="60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf" Jul 24 16:08:27.842634 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.842618 2560 scope.go:117] "RemoveContainer" containerID="60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf" Jul 24 16:08:27.842889 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:08:27.842871 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf\": container with ID starting with 60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf not found: ID does not exist" containerID="60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf" Jul 24 16:08:27.842935 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.842898 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf"} err="failed to get container status \"60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf\": rpc error: code = NotFound desc = could not find container \"60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf\": container with ID starting with 60578a3d3d76112393b3c10d433e9765899e8725e491ee5c60b08916ade3b8bf not found: ID does not exist" Jul 24 16:08:27.856707 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.856684 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87f758789-jc6mj"] Jul 24 16:08:27.858739 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:27.858718 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-87f758789-jc6mj"] Jul 24 16:08:29.165955 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:29.165911 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61cbab77-e0a5-48d3-97bc-7e3af545a45b" path="/var/lib/kubelet/pods/61cbab77-e0a5-48d3-97bc-7e3af545a45b/volumes" Jul 24 16:08:30.528813 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:30.528783 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:08:30.532508 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:30.532486 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f9844cddc-ksj4r" Jul 24 16:08:52.491894 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:52.491851 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:08:52.512975 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:52.512948 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:08:52.955878 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:08:52.955851 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:05.221439 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:05.221392 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:09:05.646138 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:05.646107 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:09:06.217824 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.217791 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 16:09:06.218192 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.218155 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="alertmanager" containerID="cri-o://fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d" gracePeriod=120 Jul 24 16:09:06.218192 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.218181 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy-metric" containerID="cri-o://5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686" gracePeriod=120 Jul 24 16:09:06.218414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.218204 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy-web" containerID="cri-o://4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa" gracePeriod=120 Jul 24 16:09:06.218414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.218270 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy" containerID="cri-o://06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b" gracePeriod=120 Jul 24 16:09:06.218414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.218309 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="config-reloader" containerID="cri-o://7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090" gracePeriod=120 Jul 24 16:09:06.218414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.218331 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="prom-label-proxy" containerID="cri-o://3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb" gracePeriod=120 Jul 24 16:09:06.998123 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998087 2560 generic.go:356] "Generic (PLEG): container finished" podID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerID="3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb" exitCode=0 Jul 24 16:09:06.998123 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998112 2560 generic.go:356] "Generic (PLEG): container finished" podID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerID="5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686" exitCode=0 Jul 24 16:09:06.998123 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998118 2560 generic.go:356] "Generic (PLEG): container finished" podID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerID="06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b" exitCode=0 Jul 24 16:09:06.998123 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998123 2560 generic.go:356] "Generic (PLEG): container finished" podID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerID="7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090" exitCode=0 Jul 24 16:09:06.998123 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998128 2560 generic.go:356] "Generic (PLEG): container finished" podID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerID="fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d" exitCode=0 Jul 24 16:09:06.998602 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998171 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerDied","Data":"3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb"} Jul 24 16:09:06.998602 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998260 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerDied","Data":"5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686"} Jul 24 16:09:06.998602 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998286 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerDied","Data":"06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b"} Jul 24 16:09:06.998602 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998311 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerDied","Data":"7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090"} Jul 24 16:09:06.998602 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:06.998336 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerDied","Data":"fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d"} Jul 24 16:09:07.468454 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.468428 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:07.564693 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.564665 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-tls-assets\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.564693 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.564700 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-metrics-client-ca\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.564900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.564723 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv8x7\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-kube-api-access-sv8x7\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.564900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.564761 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.564900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.564782 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-cluster-tls-config\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.564900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.564804 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.564900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.564832 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-web-config\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.564900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.564869 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-web\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.565247 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.565079 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:07.565247 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.565204 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-main-db\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.565348 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.565262 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-main-tls\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.565348 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.565316 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-volume\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.565612 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.565353 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-out\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.565802 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.565783 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-metrics-client-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.567137 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.567107 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:07.569069 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.568993 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-kube-api-access-sv8x7" (OuterVolumeSpecName: "kube-api-access-sv8x7") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "kube-api-access-sv8x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:09:07.569069 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.569045 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-out" (OuterVolumeSpecName: "config-out") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:07.569259 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.569059 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:07.569259 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.569191 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:07.569395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.569367 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:07.569498 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.569392 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:09:07.569746 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.569728 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:07.569909 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.569892 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-volume" (OuterVolumeSpecName: "config-volume") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:07.572458 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.572427 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:07.580415 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.580391 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-web-config" (OuterVolumeSpecName: "web-config") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:07.666391 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666364 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-trusted-ca-bundle\") pod \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\" (UID: \"b789ec5a-e174-4c49-8c9d-26d7b5865c94\") " Jul 24 16:09:07.666564 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666550 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666565 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-main-db\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666574 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-main-tls\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666582 2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-volume\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666591 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b789ec5a-e174-4c49-8c9d-26d7b5865c94-config-out\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666599 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-tls-assets\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666606 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv8x7\" (UniqueName: \"kubernetes.io/projected/b789ec5a-e174-4c49-8c9d-26d7b5865c94-kube-api-access-sv8x7\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666614 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666624 2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-cluster-tls-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666633 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666641 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b789ec5a-e174-4c49-8c9d-26d7b5865c94-web-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:07.666988 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.666769 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b789ec5a-e174-4c49-8c9d-26d7b5865c94" (UID: "b789ec5a-e174-4c49-8c9d-26d7b5865c94"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:07.767988 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:07.767955 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b789ec5a-e174-4c49-8c9d-26d7b5865c94-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:08.005620 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.005588 2560 generic.go:356] "Generic (PLEG): container finished" podID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerID="4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa" exitCode=0 Jul 24 16:09:08.006049 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.005643 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerDied","Data":"4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa"} Jul 24 16:09:08.006049 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.005664 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.006049 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.005720 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b789ec5a-e174-4c49-8c9d-26d7b5865c94","Type":"ContainerDied","Data":"1644695a0276e5370e133b1d439b6f5042cecc26cf5bb723fdb4155d916d1eae"} Jul 24 16:09:08.006049 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.005780 2560 scope.go:117] "RemoveContainer" containerID="3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb" Jul 24 16:09:08.014259 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.014242 2560 scope.go:117] "RemoveContainer" containerID="5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686" Jul 24 16:09:08.023309 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.023292 2560 scope.go:117] "RemoveContainer" containerID="06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b" Jul 24 16:09:08.030299 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.030285 2560 scope.go:117] "RemoveContainer" containerID="4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa" Jul 24 16:09:08.032640 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.032617 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 16:09:08.037967 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.037929 2560 scope.go:117] "RemoveContainer" containerID="7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090" Jul 24 16:09:08.040244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.040225 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 16:09:08.045456 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.045439 2560 scope.go:117] "RemoveContainer" containerID="fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d" Jul 24 16:09:08.052406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.052392 2560 scope.go:117] "RemoveContainer" containerID="e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e" Jul 24 16:09:08.058998 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.058983 2560 scope.go:117] "RemoveContainer" containerID="3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb" Jul 24 16:09:08.059269 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:08.059252 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb\": container with ID starting with 3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb not found: ID does not exist" containerID="3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb" Jul 24 16:09:08.059320 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.059280 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb"} err="failed to get container status \"3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb\": rpc error: code = NotFound desc = could not find container \"3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb\": container with ID starting with 3bfda9c1545078bfb0034ff7748fd3c9eaa17b0f6b552deb5fc8d1d3d0cd38cb not found: ID does not exist" Jul 24 16:09:08.059320 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.059298 2560 scope.go:117] "RemoveContainer" containerID="5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686" Jul 24 16:09:08.059515 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:08.059499 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686\": container with ID starting with 5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686 not found: ID does not exist" containerID="5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686" Jul 24 16:09:08.059557 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.059521 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686"} err="failed to get container status \"5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686\": rpc error: code = NotFound desc = could not find container \"5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686\": container with ID starting with 5c33072cd5a0e8110fe324410960e9c5ad7ac78737fda500daa5bae7b37fc686 not found: ID does not exist" Jul 24 16:09:08.059557 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.059536 2560 scope.go:117] "RemoveContainer" containerID="06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b" Jul 24 16:09:08.059734 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:08.059719 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b\": container with ID starting with 06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b not found: ID does not exist" containerID="06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b" Jul 24 16:09:08.059774 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.059737 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b"} err="failed to get container status \"06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b\": rpc error: code = NotFound desc = could not find container \"06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b\": container with ID starting with 06ca0260a5ea75943a912e1ca2f9a38d6f96f56dbca1c856ce656cbd502bee7b not found: ID does not exist" Jul 24 16:09:08.059774 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.059750 2560 scope.go:117] "RemoveContainer" containerID="4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa" Jul 24 16:09:08.059928 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:08.059908 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa\": container with ID starting with 4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa not found: ID does not exist" containerID="4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa" Jul 24 16:09:08.059968 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.059933 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa"} err="failed to get container status \"4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa\": rpc error: code = NotFound desc = could not find container \"4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa\": container with ID starting with 4f1ba7ca960e3dff65a98a3a76bd950e3eb9909ffc9a0226b19223d2bdb954fa not found: ID does not exist" Jul 24 16:09:08.059968 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.059945 2560 scope.go:117] "RemoveContainer" containerID="7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090" Jul 24 16:09:08.060166 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:08.060134 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090\": container with ID starting with 7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090 not found: ID does not exist" containerID="7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090" Jul 24 16:09:08.060205 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.060171 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090"} err="failed to get container status \"7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090\": rpc error: code = NotFound desc = could not find container \"7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090\": container with ID starting with 7cc11fd4eb47d356334cafca50581676e3e62acb9bbdf6c353ed799d7d6c6090 not found: ID does not exist" Jul 24 16:09:08.060205 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.060184 2560 scope.go:117] "RemoveContainer" containerID="fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d" Jul 24 16:09:08.060424 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:08.060399 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d\": container with ID starting with fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d not found: ID does not exist" containerID="fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d" Jul 24 16:09:08.060465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.060431 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d"} err="failed to get container status \"fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d\": rpc error: code = NotFound desc = could not find container \"fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d\": container with ID starting with fcdb6099a9126aa7ce9940dabd4773e096f1474af1c04cf9c6186e480c917b7d not found: ID does not exist" Jul 24 16:09:08.060465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.060445 2560 scope.go:117] "RemoveContainer" containerID="e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e" Jul 24 16:09:08.060667 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:08.060651 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e\": container with ID starting with e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e not found: ID does not exist" containerID="e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e" Jul 24 16:09:08.060700 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.060670 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e"} err="failed to get container status \"e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e\": rpc error: code = NotFound desc = could not find container \"e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e\": container with ID starting with e88f9d6560bf69683f54b10fffbdb70834eed11f4e07269706ba36631d64936e not found: ID does not exist" Jul 24 16:09:08.085128 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085104 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 16:09:08.085493 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085478 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="config-reloader" Jul 24 16:09:08.085539 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085500 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="config-reloader" Jul 24 16:09:08.085539 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085510 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy-web" Jul 24 16:09:08.085539 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085516 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy-web" Jul 24 16:09:08.085539 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085526 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="init-config-reloader" Jul 24 16:09:08.085539 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085531 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="init-config-reloader" Jul 24 16:09:08.085539 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085538 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61cbab77-e0a5-48d3-97bc-7e3af545a45b" containerName="console" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085544 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="61cbab77-e0a5-48d3-97bc-7e3af545a45b" containerName="console" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085552 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085557 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085566 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727735e9-6b62-45d0-adca-6b988fe783b9" containerName="console" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085575 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="727735e9-6b62-45d0-adca-6b988fe783b9" containerName="console" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085583 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy-metric" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085588 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy-metric" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085598 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="alertmanager" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085603 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="alertmanager" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085607 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="prom-label-proxy" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085612 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="prom-label-proxy" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085672 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="727735e9-6b62-45d0-adca-6b988fe783b9" containerName="console" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085682 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="prom-label-proxy" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085687 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="61cbab77-e0a5-48d3-97bc-7e3af545a45b" containerName="console" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085693 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy-web" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085700 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy-metric" Jul 24 16:09:08.085699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085706 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="alertmanager" Jul 24 16:09:08.086177 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085712 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="kube-rbac-proxy" Jul 24 16:09:08.086177 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.085718 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" containerName="config-reloader" Jul 24 16:09:08.089774 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.089758 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.095691 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.095671 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 16:09:08.096112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.096091 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 16:09:08.096112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.096112 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 16:09:08.096304 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.096223 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 16:09:08.096587 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.096371 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 16:09:08.096587 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.096579 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 16:09:08.096871 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.096850 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 16:09:08.096952 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.096893 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 16:09:08.096952 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.096899 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pszxv\"" Jul 24 16:09:08.102502 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.102480 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 16:09:08.122749 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.122726 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 16:09:08.170161 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170121 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-web-config\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170279 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170168 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170279 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170190 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c527801d-3221-40bc-8acf-42cd957d008c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170279 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170235 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c527801d-3221-40bc-8acf-42cd957d008c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170279 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170263 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170409 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170321 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170409 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170343 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c527801d-3221-40bc-8acf-42cd957d008c-config-out\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170409 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170365 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-config-volume\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170409 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170392 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170549 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170461 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c527801d-3221-40bc-8acf-42cd957d008c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170549 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170512 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqfr\" (UniqueName: \"kubernetes.io/projected/c527801d-3221-40bc-8acf-42cd957d008c-kube-api-access-9wqfr\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170633 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170552 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.170633 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.170577 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c527801d-3221-40bc-8acf-42cd957d008c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271318 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271230 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271318 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271266 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c527801d-3221-40bc-8acf-42cd957d008c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271318 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271292 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c527801d-3221-40bc-8acf-42cd957d008c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271402 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271438 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271458 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c527801d-3221-40bc-8acf-42cd957d008c-config-out\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271476 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-config-volume\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271520 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271652 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c527801d-3221-40bc-8acf-42cd957d008c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271694 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c527801d-3221-40bc-8acf-42cd957d008c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271742 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqfr\" (UniqueName: \"kubernetes.io/projected/c527801d-3221-40bc-8acf-42cd957d008c-kube-api-access-9wqfr\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271777 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.271825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c527801d-3221-40bc-8acf-42cd957d008c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.272109 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.271847 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-web-config\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.272346 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.272324 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c527801d-3221-40bc-8acf-42cd957d008c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.274789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.274571 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c527801d-3221-40bc-8acf-42cd957d008c-config-out\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.274789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.274624 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-config-volume\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.274789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.274664 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.274789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.274719 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.274789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.274738 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.275053 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.274819 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-web-config\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.275095 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.275082 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c527801d-3221-40bc-8acf-42cd957d008c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.275175 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.275136 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.275378 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.275360 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c527801d-3221-40bc-8acf-42cd957d008c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.276554 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.276534 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c527801d-3221-40bc-8acf-42cd957d008c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.281352 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.281336 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqfr\" (UniqueName: \"kubernetes.io/projected/c527801d-3221-40bc-8acf-42cd957d008c-kube-api-access-9wqfr\") pod \"alertmanager-main-0\" (UID: \"c527801d-3221-40bc-8acf-42cd957d008c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.401360 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.401319 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 16:09:08.555847 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:08.555826 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 16:09:08.557897 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:09:08.557870 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc527801d_3221_40bc_8acf_42cd957d008c.slice/crio-1acbbdaa9a77b8be9cd4dc0f8a34fb3d4da549d8167f8b71684773b9ba0a928f WatchSource:0}: Error finding container 1acbbdaa9a77b8be9cd4dc0f8a34fb3d4da549d8167f8b71684773b9ba0a928f: Status 404 returned error can't find the container with id 1acbbdaa9a77b8be9cd4dc0f8a34fb3d4da549d8167f8b71684773b9ba0a928f Jul 24 16:09:09.016594 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:09.016559 2560 generic.go:356] "Generic (PLEG): container finished" podID="c527801d-3221-40bc-8acf-42cd957d008c" containerID="e0795caabaafb83f12a21d41bdc665abfd04f37d4dafee2a6c6e72e6e6fa5b77" exitCode=0 Jul 24 16:09:09.017067 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:09.016652 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c527801d-3221-40bc-8acf-42cd957d008c","Type":"ContainerDied","Data":"e0795caabaafb83f12a21d41bdc665abfd04f37d4dafee2a6c6e72e6e6fa5b77"} Jul 24 16:09:09.017067 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:09.016774 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c527801d-3221-40bc-8acf-42cd957d008c","Type":"ContainerStarted","Data":"1acbbdaa9a77b8be9cd4dc0f8a34fb3d4da549d8167f8b71684773b9ba0a928f"} Jul 24 16:09:09.167200 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:09.167175 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b789ec5a-e174-4c49-8c9d-26d7b5865c94" path="/var/lib/kubelet/pods/b789ec5a-e174-4c49-8c9d-26d7b5865c94/volumes" Jul 24 16:09:10.024102 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.024065 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c527801d-3221-40bc-8acf-42cd957d008c","Type":"ContainerStarted","Data":"0adcbc08e6afb0eee7ea4f8f7463052c7362323e9e8f4dd75c922ccc2f2ab062"} Jul 24 16:09:10.024488 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.024166 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c527801d-3221-40bc-8acf-42cd957d008c","Type":"ContainerStarted","Data":"d95b6f5676a71ffd5913fa2ad034fcd7230c52f53ee6542caf14fc482315252a"} Jul 24 16:09:10.024488 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.024211 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c527801d-3221-40bc-8acf-42cd957d008c","Type":"ContainerStarted","Data":"de0670d14bcd9756ab4a46d79ecd7d21404d0db28801cc7b29d13fa0b489814f"} Jul 24 16:09:10.024488 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.024248 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c527801d-3221-40bc-8acf-42cd957d008c","Type":"ContainerStarted","Data":"1ffa0aede10e580ce540cf83f43b1bed88b5a89cf7edff2f5beff05b8cc965a3"} Jul 24 16:09:10.024488 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.024283 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c527801d-3221-40bc-8acf-42cd957d008c","Type":"ContainerStarted","Data":"18e854b50166bcadf3a4e44dcdb0d8d7189db7511a22ef8c9f4b324c4a97e516"} Jul 24 16:09:10.024488 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.024334 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c527801d-3221-40bc-8acf-42cd957d008c","Type":"ContainerStarted","Data":"4bf01d6c7cbc1e3c4c7f07b15b3d21e643bbd581618e57c1fc5738cbf3b41e13"} Jul 24 16:09:10.059639 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.059587 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.04274842 podStartE2EDuration="2.059565128s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.816708ms" isStatefulPod=false podCreationTimestamp="2026-07-24 16:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:09:10.058388595 +0000 UTC m=+169.490023955" watchObservedRunningTime="2026-07-24 16:09:10.059565128 +0000 UTC m=+169.491200675" Jul 24 16:09:10.630101 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.630064 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 16:09:10.630412 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.630370 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="prometheus" containerID="cri-o://f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" gracePeriod=600 Jul 24 16:09:10.630412 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.630394 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy" containerID="cri-o://bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" gracePeriod=600 Jul 24 16:09:10.630607 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.630420 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="thanos-sidecar" containerID="cri-o://ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" gracePeriod=600 Jul 24 16:09:10.630607 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.630467 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="config-reloader" containerID="cri-o://363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" gracePeriod=600 Jul 24 16:09:10.630607 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.630450 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" gracePeriod=600 Jul 24 16:09:10.630607 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.630577 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy-web" containerID="cri-o://39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" gracePeriod=600 Jul 24 16:09:10.888030 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.887974 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:10.895749 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895729 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-kube-rbac-proxy\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.895862 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895764 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-kubelet-serving-ca-bundle\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.895862 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895791 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-1\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.895862 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895807 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-web-config\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.895862 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895840 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg5c2\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-kube-api-access-bg5c2\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.895862 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895859 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-metrics-client-ca\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895885 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-tls\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895910 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-tls-assets\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895932 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-grpc-tls\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895965 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-serving-certs-ca-bundle\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.895994 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config-out\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896023 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896051 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-2\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896075 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896102 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-0\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896103 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896160 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-trusted-ca-bundle\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896139 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896234 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-db\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896266 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-thanos-prometheus-http-client-file\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896304 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-metrics-client-certs\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896359 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\" (UID: \"5a083fae-642e-4b16-acf9-f9c5d721f5cc\") " Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896399 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896453 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896692 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896712 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896728 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.896773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.896742 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.897384 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.897254 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:10.898132 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.898103 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:10.898729 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.898701 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:10.899632 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.899604 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:10.899722 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.899659 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:10.899924 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.899887 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:10.901612 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.900803 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:10.901612 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.900954 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config-out" (OuterVolumeSpecName: "config-out") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:10.901612 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.901055 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-kube-api-access-bg5c2" (OuterVolumeSpecName: "kube-api-access-bg5c2") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "kube-api-access-bg5c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:09:10.902302 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.902252 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:09:10.902302 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.902288 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:10.902458 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.902370 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:10.903654 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.903621 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:10.904032 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.904011 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config" (OuterVolumeSpecName: "config") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:10.904380 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.904364 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:10.913620 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.913599 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-web-config" (OuterVolumeSpecName: "web-config") pod "5a083fae-642e-4b16-acf9-f9c5d721f5cc" (UID: "5a083fae-642e-4b16-acf9-f9c5d721f5cc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:10.997255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997222 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-db\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997252 2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997263 2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-metrics-client-certs\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997272 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997281 2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-kube-rbac-proxy\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997290 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-web-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997298 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg5c2\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-kube-api-access-bg5c2\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997307 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-configmap-metrics-client-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997315 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997322 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a083fae-642e-4b16-acf9-f9c5d721f5cc-tls-assets\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997330 2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-grpc-tls\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997359 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config-out\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997367 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997376 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997384 2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a083fae-642e-4b16-acf9-f9c5d721f5cc-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:10.997472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:10.997391 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a083fae-642e-4b16-acf9-f9c5d721f5cc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:11.030375 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030338 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerID="74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" exitCode=0 Jul 24 16:09:11.030375 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030365 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerID="bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" exitCode=0 Jul 24 16:09:11.030375 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030373 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerID="39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" exitCode=0 Jul 24 16:09:11.030375 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030380 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerID="ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" exitCode=0 Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030387 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerID="363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" exitCode=0 Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030394 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerID="f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" exitCode=0 Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030404 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerDied","Data":"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21"} Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030436 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030514 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerDied","Data":"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6"} Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030559 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerDied","Data":"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94"} Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030601 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerDied","Data":"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2"} Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030590 2560 scope.go:117] "RemoveContainer" containerID="74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030642 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerDied","Data":"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1"} Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030685 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerDied","Data":"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646"} Jul 24 16:09:11.030864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.030724 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a083fae-642e-4b16-acf9-f9c5d721f5cc","Type":"ContainerDied","Data":"7b9f3fd7ba54bddb438933f0c679f0c9808d4971b19db3896155b0db699670f7"} Jul 24 16:09:11.038969 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.038941 2560 scope.go:117] "RemoveContainer" containerID="bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" Jul 24 16:09:11.046130 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.046116 2560 scope.go:117] "RemoveContainer" containerID="39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" Jul 24 16:09:11.055137 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.055112 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 16:09:11.056029 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.056009 2560 scope.go:117] "RemoveContainer" containerID="ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" Jul 24 16:09:11.059830 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.059806 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 16:09:11.063921 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.063900 2560 scope.go:117] "RemoveContainer" containerID="363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" Jul 24 16:09:11.070924 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.070903 2560 scope.go:117] "RemoveContainer" containerID="f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" Jul 24 16:09:11.077973 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.077956 2560 scope.go:117] "RemoveContainer" containerID="1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9" Jul 24 16:09:11.084960 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.084942 2560 scope.go:117] "RemoveContainer" containerID="74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" Jul 24 16:09:11.085292 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:11.085275 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": container with ID starting with 74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21 not found: ID does not exist" containerID="74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" Jul 24 16:09:11.085362 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.085300 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21"} err="failed to get container status \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": rpc error: code = NotFound desc = could not find container \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": container with ID starting with 74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21 not found: ID does not exist" Jul 24 16:09:11.085362 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.085319 2560 scope.go:117] "RemoveContainer" containerID="bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" Jul 24 16:09:11.085556 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:11.085534 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": container with ID starting with bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6 not found: ID does not exist" containerID="bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" Jul 24 16:09:11.085621 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.085565 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6"} err="failed to get container status \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": rpc error: code = NotFound desc = could not find container \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": container with ID starting with bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6 not found: ID does not exist" Jul 24 16:09:11.085621 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.085588 2560 scope.go:117] "RemoveContainer" containerID="39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" Jul 24 16:09:11.085845 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:11.085829 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": container with ID starting with 39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94 not found: ID does not exist" containerID="39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" Jul 24 16:09:11.085881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.085851 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94"} err="failed to get container status \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": rpc error: code = NotFound desc = could not find container \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": container with ID starting with 39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94 not found: ID does not exist" Jul 24 16:09:11.085881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.085865 2560 scope.go:117] "RemoveContainer" containerID="ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" Jul 24 16:09:11.086089 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:11.086073 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": container with ID starting with ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2 not found: ID does not exist" containerID="ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" Jul 24 16:09:11.086126 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.086092 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2"} err="failed to get container status \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": rpc error: code = NotFound desc = could not find container \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": container with ID starting with ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2 not found: ID does not exist" Jul 24 16:09:11.086126 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.086107 2560 scope.go:117] "RemoveContainer" containerID="363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" Jul 24 16:09:11.086369 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:11.086352 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": container with ID starting with 363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1 not found: ID does not exist" containerID="363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" Jul 24 16:09:11.086436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.086376 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1"} err="failed to get container status \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": rpc error: code = NotFound desc = could not find container \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": container with ID starting with 363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1 not found: ID does not exist" Jul 24 16:09:11.086436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.086398 2560 scope.go:117] "RemoveContainer" containerID="f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" Jul 24 16:09:11.086630 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:11.086614 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": container with ID starting with f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646 not found: ID does not exist" containerID="f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" Jul 24 16:09:11.086694 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.086636 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646"} err="failed to get container status \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": rpc error: code = NotFound desc = could not find container \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": container with ID starting with f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646 not found: ID does not exist" Jul 24 16:09:11.086694 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.086655 2560 scope.go:117] "RemoveContainer" containerID="1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9" Jul 24 16:09:11.086901 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:11.086884 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": container with ID starting with 1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9 not found: ID does not exist" containerID="1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9" Jul 24 16:09:11.086938 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.086906 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9"} err="failed to get container status \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": rpc error: code = NotFound desc = could not find container \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": container with ID starting with 1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9 not found: ID does not exist" Jul 24 16:09:11.086938 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.086920 2560 scope.go:117] "RemoveContainer" containerID="74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" Jul 24 16:09:11.087128 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087111 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21"} err="failed to get container status \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": rpc error: code = NotFound desc = could not find container \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": container with ID starting with 74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21 not found: ID does not exist" Jul 24 16:09:11.087232 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087130 2560 scope.go:117] "RemoveContainer" containerID="bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" Jul 24 16:09:11.087369 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087352 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6"} err="failed to get container status \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": rpc error: code = NotFound desc = could not find container \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": container with ID starting with bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6 not found: ID does not exist" Jul 24 16:09:11.087431 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087371 2560 scope.go:117] "RemoveContainer" containerID="39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" Jul 24 16:09:11.087550 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087533 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94"} err="failed to get container status \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": rpc error: code = NotFound desc = could not find container \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": container with ID starting with 39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94 not found: ID does not exist" Jul 24 16:09:11.087601 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087551 2560 scope.go:117] "RemoveContainer" containerID="ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" Jul 24 16:09:11.087719 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087703 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2"} err="failed to get container status \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": rpc error: code = NotFound desc = could not find container \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": container with ID starting with ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2 not found: ID does not exist" Jul 24 16:09:11.087761 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087722 2560 scope.go:117] "RemoveContainer" containerID="363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" Jul 24 16:09:11.087907 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087891 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1"} err="failed to get container status \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": rpc error: code = NotFound desc = could not find container \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": container with ID starting with 363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1 not found: ID does not exist" Jul 24 16:09:11.087973 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.087909 2560 scope.go:117] "RemoveContainer" containerID="f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" Jul 24 16:09:11.088170 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088130 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646"} err="failed to get container status \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": rpc error: code = NotFound desc = could not find container \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": container with ID starting with f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646 not found: ID does not exist" Jul 24 16:09:11.088228 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088169 2560 scope.go:117] "RemoveContainer" containerID="1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9" Jul 24 16:09:11.088372 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088354 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9"} err="failed to get container status \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": rpc error: code = NotFound desc = could not find container \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": container with ID starting with 1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9 not found: ID does not exist" Jul 24 16:09:11.088411 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088373 2560 scope.go:117] "RemoveContainer" containerID="74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" Jul 24 16:09:11.088547 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088532 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21"} err="failed to get container status \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": rpc error: code = NotFound desc = could not find container \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": container with ID starting with 74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21 not found: ID does not exist" Jul 24 16:09:11.088585 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088548 2560 scope.go:117] "RemoveContainer" containerID="bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" Jul 24 16:09:11.088734 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088715 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6"} err="failed to get container status \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": rpc error: code = NotFound desc = could not find container \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": container with ID starting with bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6 not found: ID does not exist" Jul 24 16:09:11.088818 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088735 2560 scope.go:117] "RemoveContainer" containerID="39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" Jul 24 16:09:11.088903 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088882 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94"} err="failed to get container status \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": rpc error: code = NotFound desc = could not find container \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": container with ID starting with 39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94 not found: ID does not exist" Jul 24 16:09:11.088903 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.088901 2560 scope.go:117] "RemoveContainer" containerID="ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" Jul 24 16:09:11.089157 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089125 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2"} err="failed to get container status \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": rpc error: code = NotFound desc = could not find container \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": container with ID starting with ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2 not found: ID does not exist" Jul 24 16:09:11.089211 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089162 2560 scope.go:117] "RemoveContainer" containerID="363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" Jul 24 16:09:11.089425 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089408 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1"} err="failed to get container status \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": rpc error: code = NotFound desc = could not find container \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": container with ID starting with 363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1 not found: ID does not exist" Jul 24 16:09:11.089478 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089425 2560 scope.go:117] "RemoveContainer" containerID="f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" Jul 24 16:09:11.089600 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089586 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646"} err="failed to get container status \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": rpc error: code = NotFound desc = could not find container \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": container with ID starting with f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646 not found: ID does not exist" Jul 24 16:09:11.089646 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089602 2560 scope.go:117] "RemoveContainer" containerID="1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9" Jul 24 16:09:11.089763 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089744 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9"} err="failed to get container status \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": rpc error: code = NotFound desc = could not find container \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": container with ID starting with 1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9 not found: ID does not exist" Jul 24 16:09:11.089802 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089764 2560 scope.go:117] "RemoveContainer" containerID="74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" Jul 24 16:09:11.089962 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089943 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21"} err="failed to get container status \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": rpc error: code = NotFound desc = could not find container \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": container with ID starting with 74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21 not found: ID does not exist" Jul 24 16:09:11.090001 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.089963 2560 scope.go:117] "RemoveContainer" containerID="bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" Jul 24 16:09:11.090187 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.090168 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6"} err="failed to get container status \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": rpc error: code = NotFound desc = could not find container \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": container with ID starting with bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6 not found: ID does not exist" Jul 24 16:09:11.090187 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.090187 2560 scope.go:117] "RemoveContainer" containerID="39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" Jul 24 16:09:11.090445 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.090419 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94"} err="failed to get container status \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": rpc error: code = NotFound desc = could not find container \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": container with ID starting with 39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94 not found: ID does not exist" Jul 24 16:09:11.090445 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.090443 2560 scope.go:117] "RemoveContainer" containerID="ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" Jul 24 16:09:11.090653 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.090636 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2"} err="failed to get container status \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": rpc error: code = NotFound desc = could not find container \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": container with ID starting with ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2 not found: ID does not exist" Jul 24 16:09:11.090703 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.090655 2560 scope.go:117] "RemoveContainer" containerID="363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" Jul 24 16:09:11.090833 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.090817 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1"} err="failed to get container status \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": rpc error: code = NotFound desc = could not find container \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": container with ID starting with 363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1 not found: ID does not exist" Jul 24 16:09:11.090878 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.090836 2560 scope.go:117] "RemoveContainer" containerID="f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" Jul 24 16:09:11.091011 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.090987 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646"} err="failed to get container status \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": rpc error: code = NotFound desc = could not find container \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": container with ID starting with f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646 not found: ID does not exist" Jul 24 16:09:11.091054 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.091011 2560 scope.go:117] "RemoveContainer" containerID="1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9" Jul 24 16:09:11.091206 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.091188 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9"} err="failed to get container status \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": rpc error: code = NotFound desc = could not find container \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": container with ID starting with 1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9 not found: ID does not exist" Jul 24 16:09:11.091263 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.091207 2560 scope.go:117] "RemoveContainer" containerID="74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" Jul 24 16:09:11.091431 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.091414 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21"} err="failed to get container status \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": rpc error: code = NotFound desc = could not find container \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": container with ID starting with 74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21 not found: ID does not exist" Jul 24 16:09:11.091479 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.091431 2560 scope.go:117] "RemoveContainer" containerID="bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" Jul 24 16:09:11.091635 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.091620 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6"} err="failed to get container status \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": rpc error: code = NotFound desc = could not find container \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": container with ID starting with bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6 not found: ID does not exist" Jul 24 16:09:11.091677 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.091635 2560 scope.go:117] "RemoveContainer" containerID="39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" Jul 24 16:09:11.091820 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.091803 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94"} err="failed to get container status \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": rpc error: code = NotFound desc = could not find container \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": container with ID starting with 39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94 not found: ID does not exist" Jul 24 16:09:11.091858 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.091823 2560 scope.go:117] "RemoveContainer" containerID="ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" Jul 24 16:09:11.092027 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.092011 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2"} err="failed to get container status \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": rpc error: code = NotFound desc = could not find container \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": container with ID starting with ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2 not found: ID does not exist" Jul 24 16:09:11.092070 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.092028 2560 scope.go:117] "RemoveContainer" containerID="363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" Jul 24 16:09:11.092420 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.092371 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1"} err="failed to get container status \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": rpc error: code = NotFound desc = could not find container \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": container with ID starting with 363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1 not found: ID does not exist" Jul 24 16:09:11.092420 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.092414 2560 scope.go:117] "RemoveContainer" containerID="f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" Jul 24 16:09:11.092939 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.092899 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646"} err="failed to get container status \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": rpc error: code = NotFound desc = could not find container \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": container with ID starting with f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646 not found: ID does not exist" Jul 24 16:09:11.092939 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.092923 2560 scope.go:117] "RemoveContainer" containerID="1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9" Jul 24 16:09:11.093318 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.093300 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9"} err="failed to get container status \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": rpc error: code = NotFound desc = could not find container \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": container with ID starting with 1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9 not found: ID does not exist" Jul 24 16:09:11.093398 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.093321 2560 scope.go:117] "RemoveContainer" containerID="74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21" Jul 24 16:09:11.093598 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.093576 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21"} err="failed to get container status \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": rpc error: code = NotFound desc = could not find container \"74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21\": container with ID starting with 74f32a33b3fe2b61c888a08a89cd1856f11651375f5e4a62b9c4c7ecb7672b21 not found: ID does not exist" Jul 24 16:09:11.093649 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.093601 2560 scope.go:117] "RemoveContainer" containerID="bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6" Jul 24 16:09:11.093871 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.093853 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6"} err="failed to get container status \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": rpc error: code = NotFound desc = could not find container \"bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6\": container with ID starting with bdf3f1519f0b410baf628d7959a330b6a869ab51f51dde3c51afc945d4e039d6 not found: ID does not exist" Jul 24 16:09:11.093925 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.093871 2560 scope.go:117] "RemoveContainer" containerID="39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94" Jul 24 16:09:11.094109 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.094087 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94"} err="failed to get container status \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": rpc error: code = NotFound desc = could not find container \"39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94\": container with ID starting with 39eca0079a40f4868f6d82b475e4763d95af332e5482180f5bd41e19d1a27e94 not found: ID does not exist" Jul 24 16:09:11.094168 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.094112 2560 scope.go:117] "RemoveContainer" containerID="ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2" Jul 24 16:09:11.094392 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.094372 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2"} err="failed to get container status \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": rpc error: code = NotFound desc = could not find container \"ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2\": container with ID starting with ea8a3f806701fdc24f2fc32c9712707cf737d5d2fbf947d8a4b92a00eb14c9c2 not found: ID does not exist" Jul 24 16:09:11.094443 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.094400 2560 scope.go:117] "RemoveContainer" containerID="363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1" Jul 24 16:09:11.094607 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.094592 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1"} err="failed to get container status \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": rpc error: code = NotFound desc = could not find container \"363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1\": container with ID starting with 363f08f8d63c5a2f1056c52d508d68b07ab6483d01047c606cdfcb2e83968cf1 not found: ID does not exist" Jul 24 16:09:11.094662 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.094607 2560 scope.go:117] "RemoveContainer" containerID="f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646" Jul 24 16:09:11.094702 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.094662 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 16:09:11.094849 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.094828 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646"} err="failed to get container status \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": rpc error: code = NotFound desc = could not find container \"f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646\": container with ID starting with f1686ac0670c706ffec96e0c04de88e7c7bbbee4ef21018b9cccfcbb15fcd646 not found: ID does not exist" Jul 24 16:09:11.094897 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.094851 2560 scope.go:117] "RemoveContainer" containerID="1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9" Jul 24 16:09:11.095037 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095023 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="init-config-reloader" Jul 24 16:09:11.095099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095040 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="init-config-reloader" Jul 24 16:09:11.095099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095052 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="prometheus" Jul 24 16:09:11.095099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095058 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="prometheus" Jul 24 16:09:11.095099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095065 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy-thanos" Jul 24 16:09:11.095099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095073 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy-thanos" Jul 24 16:09:11.095099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095075 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9"} err="failed to get container status \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": rpc error: code = NotFound desc = could not find container \"1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9\": container with ID starting with 1283d02820c7c3e9d68c67ada3325fc2d3ec1412fe461f183b7dc787d9170eb9 not found: ID does not exist" Jul 24 16:09:11.095099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095090 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="config-reloader" Jul 24 16:09:11.095099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095098 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="config-reloader" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095110 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy-web" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095116 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy-web" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095124 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="thanos-sidecar" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095129 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="thanos-sidecar" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095137 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095159 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095213 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy-thanos" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095221 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="config-reloader" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095228 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095235 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="thanos-sidecar" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095242 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="prometheus" Jul 24 16:09:11.095353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.095247 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" containerName="kube-rbac-proxy-web" Jul 24 16:09:11.099221 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.099202 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.108400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.106015 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 16:09:11.108400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.106273 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 16:09:11.108400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.106877 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5gq2v\"" Jul 24 16:09:11.108400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.107044 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 16:09:11.108400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.107230 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-albv6s7ruqn2n\"" Jul 24 16:09:11.108400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.107383 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 16:09:11.108400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.107871 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 16:09:11.108400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.108057 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 16:09:11.113507 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.109564 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 16:09:11.113507 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.109752 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 16:09:11.113507 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.109912 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 16:09:11.113507 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.110064 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 16:09:11.113507 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.110229 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 16:09:11.113507 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.113113 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 16:09:11.116490 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.116313 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 16:09:11.118903 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.118866 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 16:09:11.119181 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.119138 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 16:09:11.165633 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.165540 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a083fae-642e-4b16-acf9-f9c5d721f5cc" path="/var/lib/kubelet/pods/5a083fae-642e-4b16-acf9-f9c5d721f5cc/volumes" Jul 24 16:09:11.200010 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.199986 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200031 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200111 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200051 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200228 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200213 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-config\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200294 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200243 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-web-config\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200356 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200310 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17b7e2ee-207b-4fdc-9e59-b112fd10843a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200410 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200354 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17b7e2ee-207b-4fdc-9e59-b112fd10843a-config-out\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200410 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200380 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200536 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200515 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200580 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200637 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200616 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200685 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200671 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200718 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200700 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9nt\" (UniqueName: \"kubernetes.io/projected/17b7e2ee-207b-4fdc-9e59-b112fd10843a-kube-api-access-cs9nt\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200777 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200760 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200814 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200791 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200835 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.200991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.200974 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.201045 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.201008 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.201098 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.201056 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302423 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302375 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302423 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302423 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302654 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302537 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302654 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302563 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9nt\" (UniqueName: \"kubernetes.io/projected/17b7e2ee-207b-4fdc-9e59-b112fd10843a-kube-api-access-cs9nt\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302654 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302601 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302654 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302854 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302660 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302854 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302689 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302854 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302734 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.302854 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.302764 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303015 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303104 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303052 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303104 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303082 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303112 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303171 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-config\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303197 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-web-config\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17b7e2ee-207b-4fdc-9e59-b112fd10843a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303427 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303283 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17b7e2ee-207b-4fdc-9e59-b112fd10843a-config-out\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303427 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303305 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303427 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303340 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303427 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303390 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303647 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303627 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303967 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303756 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.303967 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.303957 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.304658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.304632 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.306024 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.305730 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.306024 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.305844 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.306196 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.306052 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.306926 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.306380 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-config\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.306926 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.306518 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.306926 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.306759 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.306926 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.306774 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-web-config\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.306926 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.306859 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.307265 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.307075 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17b7e2ee-207b-4fdc-9e59-b112fd10843a-config-out\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.307634 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.307607 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b7e2ee-207b-4fdc-9e59-b112fd10843a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.308170 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.308123 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.308484 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.308464 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.308699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.308685 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17b7e2ee-207b-4fdc-9e59-b112fd10843a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.308842 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.308829 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17b7e2ee-207b-4fdc-9e59-b112fd10843a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.310755 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.310740 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9nt\" (UniqueName: \"kubernetes.io/projected/17b7e2ee-207b-4fdc-9e59-b112fd10843a-kube-api-access-cs9nt\") pod \"prometheus-k8s-0\" (UID: \"17b7e2ee-207b-4fdc-9e59-b112fd10843a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.432650 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.432572 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:11.569547 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:11.569524 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 16:09:11.571275 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:09:11.571250 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b7e2ee_207b_4fdc_9e59_b112fd10843a.slice/crio-e509fab2bca6b33e99d8b74b7160890c2f728027efac21c8ae2813799d4b2372 WatchSource:0}: Error finding container e509fab2bca6b33e99d8b74b7160890c2f728027efac21c8ae2813799d4b2372: Status 404 returned error can't find the container with id e509fab2bca6b33e99d8b74b7160890c2f728027efac21c8ae2813799d4b2372 Jul 24 16:09:12.036553 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:12.036523 2560 generic.go:356] "Generic (PLEG): container finished" podID="17b7e2ee-207b-4fdc-9e59-b112fd10843a" containerID="7bf474c10b3b775335dc23d5f1d7b08d04a3dd7ff94900d58f07e55a6aadd742" exitCode=0 Jul 24 16:09:12.036916 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:12.036581 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17b7e2ee-207b-4fdc-9e59-b112fd10843a","Type":"ContainerDied","Data":"7bf474c10b3b775335dc23d5f1d7b08d04a3dd7ff94900d58f07e55a6aadd742"} Jul 24 16:09:12.036916 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:12.036628 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17b7e2ee-207b-4fdc-9e59-b112fd10843a","Type":"ContainerStarted","Data":"e509fab2bca6b33e99d8b74b7160890c2f728027efac21c8ae2813799d4b2372"} Jul 24 16:09:13.044217 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:13.044185 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17b7e2ee-207b-4fdc-9e59-b112fd10843a","Type":"ContainerStarted","Data":"6f8ff54ea62606896f5349733ea3142f9d6495f006a9589e9c92ffec8ccef76d"} Jul 24 16:09:13.044585 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:13.044289 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17b7e2ee-207b-4fdc-9e59-b112fd10843a","Type":"ContainerStarted","Data":"dc3489ee979f4b7abf38693056e4f146567bd2fe9782c26148146ad8ac354b8d"} Jul 24 16:09:13.044585 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:13.044317 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17b7e2ee-207b-4fdc-9e59-b112fd10843a","Type":"ContainerStarted","Data":"eb800bc01ac244777688757daa5ef0e9e137f06ac3de7b0bac80223ae188dfac"} Jul 24 16:09:13.044585 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:13.044349 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17b7e2ee-207b-4fdc-9e59-b112fd10843a","Type":"ContainerStarted","Data":"4a9391b13106e245e5dde1c8833b592013a6a139bc9bedb1e4c46735edc27d82"} Jul 24 16:09:13.044585 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:13.044389 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17b7e2ee-207b-4fdc-9e59-b112fd10843a","Type":"ContainerStarted","Data":"c56f4ab57a58bdef42603ba54b2ba3931bcf9171462fac8454da9d8aba884535"} Jul 24 16:09:13.044585 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:13.044415 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17b7e2ee-207b-4fdc-9e59-b112fd10843a","Type":"ContainerStarted","Data":"ea181583454e11fd1d6a1eb70165b62c749d40dc8549527c3968faca79990bc3"} Jul 24 16:09:13.079771 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:13.079713 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.05989748 podStartE2EDuration="2.079696847s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.799367ms" isStatefulPod=false podCreationTimestamp="2026-07-24 16:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:09:13.078270273 +0000 UTC m=+172.509905634" watchObservedRunningTime="2026-07-24 16:09:13.079696847 +0000 UTC m=+172.511332206" Jul 24 16:09:15.417001 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.416961 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59dcd69dc6-8js6b"] Jul 24 16:09:15.420252 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.420228 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.436321 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.436292 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59dcd69dc6-8js6b"] Jul 24 16:09:15.437718 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.437695 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-service-ca\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.437824 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.437738 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-oauth-serving-cert\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.437880 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.437824 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-trusted-ca-bundle\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.437880 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.437866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkpfg\" (UniqueName: \"kubernetes.io/projected/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-kube-api-access-vkpfg\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.437959 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.437933 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-serving-cert\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.437994 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.437980 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-config\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.438038 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.438007 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-oauth-config\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.519098 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.519067 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:09:15.538872 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.538842 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-config\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.539005 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.538877 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-oauth-config\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.539005 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.538898 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-service-ca\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.539005 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.538927 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-oauth-serving-cert\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.539005 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.538965 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-trusted-ca-bundle\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.539222 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.539006 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkpfg\" (UniqueName: \"kubernetes.io/projected/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-kube-api-access-vkpfg\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.539222 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.539056 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-serving-cert\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.539724 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.539700 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-config\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.540027 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.539972 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-service-ca\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.540239 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.540214 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-oauth-serving-cert\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.540416 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.540384 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-trusted-ca-bundle\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.541679 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.541658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-oauth-config\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.541773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.541734 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-serving-cert\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.555839 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.555821 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkpfg\" (UniqueName: \"kubernetes.io/projected/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-kube-api-access-vkpfg\") pod \"console-59dcd69dc6-8js6b\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.644089 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.644051 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:09:15.732476 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.732103 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:15.882441 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:15.882399 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59dcd69dc6-8js6b"] Jul 24 16:09:15.885184 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:09:15.885157 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52245fa_53e9_4e1d_8fa2_7befa1d9223e.slice/crio-68ae5936d1ac553f8ecd106319c0ee9fcb757ccab6c0edef702ffd682a42f88d WatchSource:0}: Error finding container 68ae5936d1ac553f8ecd106319c0ee9fcb757ccab6c0edef702ffd682a42f88d: Status 404 returned error can't find the container with id 68ae5936d1ac553f8ecd106319c0ee9fcb757ccab6c0edef702ffd682a42f88d Jul 24 16:09:16.058891 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:16.058807 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59dcd69dc6-8js6b" event={"ID":"c52245fa-53e9-4e1d-8fa2-7befa1d9223e","Type":"ContainerStarted","Data":"8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d"} Jul 24 16:09:16.058891 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:16.058858 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59dcd69dc6-8js6b" event={"ID":"c52245fa-53e9-4e1d-8fa2-7befa1d9223e","Type":"ContainerStarted","Data":"68ae5936d1ac553f8ecd106319c0ee9fcb757ccab6c0edef702ffd682a42f88d"} Jul 24 16:09:16.093948 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:16.093894 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59dcd69dc6-8js6b" podStartSLOduration=1.093877075 podStartE2EDuration="1.093877075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:09:16.091892488 +0000 UTC m=+175.523527848" watchObservedRunningTime="2026-07-24 16:09:16.093877075 +0000 UTC m=+175.525512434" Jul 24 16:09:16.433524 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:16.433487 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:09:25.733060 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:25.733030 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:25.733474 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:25.733322 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:25.737555 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:25.737537 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:26.105810 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:26.105734 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:09:26.158885 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:26.158853 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d8b4774d-qrsl2"] Jul 24 16:09:26.286476 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:26.286442 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:09:26.386134 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:26.386104 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:09:26.471590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:26.471569 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:09:32.059286 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:32.059254 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:09:51.178423 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.178361 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77d8b4774d-qrsl2" podUID="26ef84d1-6d96-4866-9eb3-72cebcb287f8" containerName="console" containerID="cri-o://e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809" gracePeriod=15 Jul 24 16:09:51.414390 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.414368 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:09:51.552454 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.552351 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-oauth-config\") pod \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " Jul 24 16:09:51.552454 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.552399 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-trusted-ca-bundle\") pod \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " Jul 24 16:09:51.552454 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.552433 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-serving-cert\") pod \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " Jul 24 16:09:51.552757 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.552491 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-oauth-serving-cert\") pod \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " Jul 24 16:09:51.552757 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.552532 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7877d\" (UniqueName: \"kubernetes.io/projected/26ef84d1-6d96-4866-9eb3-72cebcb287f8-kube-api-access-7877d\") pod \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " Jul 24 16:09:51.552757 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.552580 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-service-ca\") pod \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " Jul 24 16:09:51.552757 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.552606 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-config\") pod \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\" (UID: \"26ef84d1-6d96-4866-9eb3-72cebcb287f8\") " Jul 24 16:09:51.552945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.552847 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "26ef84d1-6d96-4866-9eb3-72cebcb287f8" (UID: "26ef84d1-6d96-4866-9eb3-72cebcb287f8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:51.552945 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.552872 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "26ef84d1-6d96-4866-9eb3-72cebcb287f8" (UID: "26ef84d1-6d96-4866-9eb3-72cebcb287f8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:51.553091 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.553073 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-config" (OuterVolumeSpecName: "console-config") pod "26ef84d1-6d96-4866-9eb3-72cebcb287f8" (UID: "26ef84d1-6d96-4866-9eb3-72cebcb287f8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:51.553335 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.553312 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-service-ca" (OuterVolumeSpecName: "service-ca") pod "26ef84d1-6d96-4866-9eb3-72cebcb287f8" (UID: "26ef84d1-6d96-4866-9eb3-72cebcb287f8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:51.554698 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.554679 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "26ef84d1-6d96-4866-9eb3-72cebcb287f8" (UID: "26ef84d1-6d96-4866-9eb3-72cebcb287f8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:51.554789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.554721 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "26ef84d1-6d96-4866-9eb3-72cebcb287f8" (UID: "26ef84d1-6d96-4866-9eb3-72cebcb287f8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:51.554844 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.554794 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ef84d1-6d96-4866-9eb3-72cebcb287f8-kube-api-access-7877d" (OuterVolumeSpecName: "kube-api-access-7877d") pod "26ef84d1-6d96-4866-9eb3-72cebcb287f8" (UID: "26ef84d1-6d96-4866-9eb3-72cebcb287f8"). InnerVolumeSpecName "kube-api-access-7877d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:09:51.653415 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.653385 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:51.653415 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.653410 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:51.653415 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.653421 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-oauth-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:51.653621 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.653430 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-trusted-ca-bundle\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:51.653621 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.653439 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26ef84d1-6d96-4866-9eb3-72cebcb287f8-console-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:51.653621 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.653450 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26ef84d1-6d96-4866-9eb3-72cebcb287f8-oauth-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:51.653621 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:51.653458 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7877d\" (UniqueName: \"kubernetes.io/projected/26ef84d1-6d96-4866-9eb3-72cebcb287f8-kube-api-access-7877d\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:09:52.208584 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:52.208553 2560 generic.go:356] "Generic (PLEG): container finished" podID="26ef84d1-6d96-4866-9eb3-72cebcb287f8" containerID="e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809" exitCode=0 Jul 24 16:09:52.209014 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:52.208606 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d8b4774d-qrsl2" Jul 24 16:09:52.209014 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:52.208644 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d8b4774d-qrsl2" event={"ID":"26ef84d1-6d96-4866-9eb3-72cebcb287f8","Type":"ContainerDied","Data":"e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809"} Jul 24 16:09:52.209014 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:52.208691 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d8b4774d-qrsl2" event={"ID":"26ef84d1-6d96-4866-9eb3-72cebcb287f8","Type":"ContainerDied","Data":"1b10200cb22aa92e78a2d016b36c42fd654c4ecf96cf441a9b591573ed6594ce"} Jul 24 16:09:52.209014 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:52.208717 2560 scope.go:117] "RemoveContainer" containerID="e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809" Jul 24 16:09:52.217413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:52.217399 2560 scope.go:117] "RemoveContainer" containerID="e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809" Jul 24 16:09:52.217647 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:09:52.217629 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809\": container with ID starting with e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809 not found: ID does not exist" containerID="e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809" Jul 24 16:09:52.217686 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:52.217656 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809"} err="failed to get container status \"e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809\": rpc error: code = NotFound desc = could not find container \"e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809\": container with ID starting with e20c672051b01b222e22bee5bdc53b94c9b9036b7e12ed112d4a2fe94eb03809 not found: ID does not exist" Jul 24 16:09:52.229944 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:52.229918 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d8b4774d-qrsl2"] Jul 24 16:09:52.234272 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:52.234253 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77d8b4774d-qrsl2"] Jul 24 16:09:53.166128 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:09:53.166092 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ef84d1-6d96-4866-9eb3-72cebcb287f8" path="/var/lib/kubelet/pods/26ef84d1-6d96-4866-9eb3-72cebcb287f8/volumes" Jul 24 16:10:11.433660 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:11.433629 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:10:11.449651 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:11.449628 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:10:12.311965 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:12.311940 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 16:10:20.474569 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:20.474531 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:20.537248 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:20.537214 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:31.381678 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.381641 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f95b8b676-pp5n9"] Jul 24 16:10:31.382125 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.381962 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ef84d1-6d96-4866-9eb3-72cebcb287f8" containerName="console" Jul 24 16:10:31.382125 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.381973 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ef84d1-6d96-4866-9eb3-72cebcb287f8" containerName="console" Jul 24 16:10:31.382125 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.382029 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ef84d1-6d96-4866-9eb3-72cebcb287f8" containerName="console" Jul 24 16:10:31.386193 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.386173 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.396804 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.396777 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f95b8b676-pp5n9"] Jul 24 16:10:31.447780 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.447746 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:31.493591 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.493550 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-serving-cert\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.493591 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.493612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-service-ca\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.493819 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.493685 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-trusted-ca-bundle\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.493819 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.493723 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-oauth-serving-cert\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.493819 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.493805 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-console-config\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.494199 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.493837 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-oauth-config\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.494199 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.493856 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29dlr\" (UniqueName: \"kubernetes.io/projected/59813405-ba3f-4f3f-832c-17d807ee62d1-kube-api-access-29dlr\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.550351 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.550327 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:31.594378 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.594351 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-serving-cert\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.594502 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.594419 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-service-ca\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.594502 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.594460 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-trusted-ca-bundle\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.594502 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.594496 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-oauth-serving-cert\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.594611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.594558 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-console-config\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.594611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.594597 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-oauth-config\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.594804 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.594782 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29dlr\" (UniqueName: \"kubernetes.io/projected/59813405-ba3f-4f3f-832c-17d807ee62d1-kube-api-access-29dlr\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.595321 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.595295 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-service-ca\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.595414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.595336 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-console-config\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.595414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.595352 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-trusted-ca-bundle\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.595490 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.595470 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-oauth-serving-cert\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.597027 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.597009 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-serving-cert\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.597453 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.597435 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-oauth-config\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.603737 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.603715 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29dlr\" (UniqueName: \"kubernetes.io/projected/59813405-ba3f-4f3f-832c-17d807ee62d1-kube-api-access-29dlr\") pod \"console-7f95b8b676-pp5n9\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.694835 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.694772 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:31.825626 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:31.825599 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f95b8b676-pp5n9"] Jul 24 16:10:31.828089 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:10:31.828056 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59813405_ba3f_4f3f_832c_17d807ee62d1.slice/crio-11dd3855011f1005d3a52188f88375b9f126ef4cd1a1396e78c336070eafc69b WatchSource:0}: Error finding container 11dd3855011f1005d3a52188f88375b9f126ef4cd1a1396e78c336070eafc69b: Status 404 returned error can't find the container with id 11dd3855011f1005d3a52188f88375b9f126ef4cd1a1396e78c336070eafc69b Jul 24 16:10:32.374051 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:32.374019 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f95b8b676-pp5n9" event={"ID":"59813405-ba3f-4f3f-832c-17d807ee62d1","Type":"ContainerStarted","Data":"dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81"} Jul 24 16:10:32.374280 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:32.374073 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f95b8b676-pp5n9" event={"ID":"59813405-ba3f-4f3f-832c-17d807ee62d1","Type":"ContainerStarted","Data":"11dd3855011f1005d3a52188f88375b9f126ef4cd1a1396e78c336070eafc69b"} Jul 24 16:10:32.394239 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:32.394193 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f95b8b676-pp5n9" podStartSLOduration=1.394178767 podStartE2EDuration="1.394178767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:10:32.392746718 +0000 UTC m=+251.824382083" watchObservedRunningTime="2026-07-24 16:10:32.394178767 +0000 UTC m=+251.825814127" Jul 24 16:10:32.495558 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:32.495525 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:32.596108 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:32.596072 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:33.398951 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:33.398918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:33.462517 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:33.462487 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:34.949850 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:34.949816 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:35.018340 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:35.018310 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:41.695871 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:41.695800 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:41.695871 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:41.695870 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:41.700335 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:41.700312 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:42.419793 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:42.419766 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:10:42.470232 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:42.470198 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59dcd69dc6-8js6b"] Jul 24 16:10:42.659628 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:42.659594 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:42.751384 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:42.751305 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:49.965797 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:49.965725 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:10:50.080320 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:10:50.080293 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:11:07.489916 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.489847 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59dcd69dc6-8js6b" podUID="c52245fa-53e9-4e1d-8fa2-7befa1d9223e" containerName="console" containerID="cri-o://8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d" gracePeriod=15 Jul 24 16:11:07.730155 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.730126 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:11:07.798930 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.798854 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-oauth-config\") pod \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " Jul 24 16:11:07.798930 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.798917 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-oauth-serving-cert\") pod \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " Jul 24 16:11:07.799136 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.798942 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-serving-cert\") pod \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " Jul 24 16:11:07.799136 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.798988 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-service-ca\") pod \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " Jul 24 16:11:07.799136 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.799027 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkpfg\" (UniqueName: \"kubernetes.io/projected/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-kube-api-access-vkpfg\") pod \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " Jul 24 16:11:07.799136 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.799053 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-config\") pod \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " Jul 24 16:11:07.799136 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.799090 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-trusted-ca-bundle\") pod \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\" (UID: \"c52245fa-53e9-4e1d-8fa2-7befa1d9223e\") " Jul 24 16:11:07.799450 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.799404 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c52245fa-53e9-4e1d-8fa2-7befa1d9223e" (UID: "c52245fa-53e9-4e1d-8fa2-7befa1d9223e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:11:07.799450 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.799402 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-service-ca" (OuterVolumeSpecName: "service-ca") pod "c52245fa-53e9-4e1d-8fa2-7befa1d9223e" (UID: "c52245fa-53e9-4e1d-8fa2-7befa1d9223e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:11:07.799718 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.799452 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-config" (OuterVolumeSpecName: "console-config") pod "c52245fa-53e9-4e1d-8fa2-7befa1d9223e" (UID: "c52245fa-53e9-4e1d-8fa2-7befa1d9223e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:11:07.799911 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.799890 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c52245fa-53e9-4e1d-8fa2-7befa1d9223e" (UID: "c52245fa-53e9-4e1d-8fa2-7befa1d9223e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:11:07.801183 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.801139 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c52245fa-53e9-4e1d-8fa2-7befa1d9223e" (UID: "c52245fa-53e9-4e1d-8fa2-7befa1d9223e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:11:07.801298 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.801278 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-kube-api-access-vkpfg" (OuterVolumeSpecName: "kube-api-access-vkpfg") pod "c52245fa-53e9-4e1d-8fa2-7befa1d9223e" (UID: "c52245fa-53e9-4e1d-8fa2-7befa1d9223e"). InnerVolumeSpecName "kube-api-access-vkpfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:11:07.801340 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.801298 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c52245fa-53e9-4e1d-8fa2-7befa1d9223e" (UID: "c52245fa-53e9-4e1d-8fa2-7befa1d9223e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:11:07.899740 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.899712 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-oauth-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:11:07.899740 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.899734 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:11:07.899893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.899747 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:11:07.899893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.899760 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkpfg\" (UniqueName: \"kubernetes.io/projected/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-kube-api-access-vkpfg\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:11:07.899893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.899773 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:11:07.899893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.899784 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-trusted-ca-bundle\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:11:07.899893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:07.899795 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c52245fa-53e9-4e1d-8fa2-7befa1d9223e-console-oauth-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:11:08.523469 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:08.523431 2560 generic.go:356] "Generic (PLEG): container finished" podID="c52245fa-53e9-4e1d-8fa2-7befa1d9223e" containerID="8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d" exitCode=0 Jul 24 16:11:08.523921 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:08.523477 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59dcd69dc6-8js6b" Jul 24 16:11:08.523921 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:08.523513 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59dcd69dc6-8js6b" event={"ID":"c52245fa-53e9-4e1d-8fa2-7befa1d9223e","Type":"ContainerDied","Data":"8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d"} Jul 24 16:11:08.523921 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:08.523561 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59dcd69dc6-8js6b" event={"ID":"c52245fa-53e9-4e1d-8fa2-7befa1d9223e","Type":"ContainerDied","Data":"68ae5936d1ac553f8ecd106319c0ee9fcb757ccab6c0edef702ffd682a42f88d"} Jul 24 16:11:08.523921 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:08.523584 2560 scope.go:117] "RemoveContainer" containerID="8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d" Jul 24 16:11:08.532005 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:08.531991 2560 scope.go:117] "RemoveContainer" containerID="8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d" Jul 24 16:11:08.532279 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:11:08.532261 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d\": container with ID starting with 8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d not found: ID does not exist" containerID="8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d" Jul 24 16:11:08.532338 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:08.532285 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d"} err="failed to get container status \"8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d\": rpc error: code = NotFound desc = could not find container \"8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d\": container with ID starting with 8b8ba2f36983a6935a08fbf8bdd45200b736d30af295c9ae97dab8236326353d not found: ID does not exist" Jul 24 16:11:08.545055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:08.545033 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59dcd69dc6-8js6b"] Jul 24 16:11:08.547043 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:08.547023 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59dcd69dc6-8js6b"] Jul 24 16:11:09.165960 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:09.165918 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52245fa-53e9-4e1d-8fa2-7befa1d9223e" path="/var/lib/kubelet/pods/c52245fa-53e9-4e1d-8fa2-7befa1d9223e/volumes" Jul 24 16:11:20.927757 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:20.927725 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:11:20.934552 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:20.934529 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:11:20.935450 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:20.935430 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 16:11:35.752163 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:35.752082 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:11:37.144350 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:37.144292 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:11:58.071734 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:58.071703 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:11:58.410644 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:11:58.410616 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:01.859442 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:01.859413 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:01.961215 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:01.961185 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:03.288592 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:03.288557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:03.405379 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:03.405346 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:03.621396 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:03.621362 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:03.800855 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:03.800822 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:05.897891 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:05.897860 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:06.324791 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:06.324719 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:07.211723 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:07.211693 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:07.313107 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:07.313075 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:08.864976 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:08.864946 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:08.951712 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:08.951678 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:10.072009 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.071978 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wnqlw"] Jul 24 16:12:10.072422 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.072324 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52245fa-53e9-4e1d-8fa2-7befa1d9223e" containerName="console" Jul 24 16:12:10.072422 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.072336 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52245fa-53e9-4e1d-8fa2-7befa1d9223e" containerName="console" Jul 24 16:12:10.072422 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.072403 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52245fa-53e9-4e1d-8fa2-7befa1d9223e" containerName="console" Jul 24 16:12:10.075336 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.075321 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:10.077934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.077916 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 16:12:10.078194 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.078178 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9c5sb\"" Jul 24 16:12:10.078942 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.078926 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 16:12:10.086560 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.086537 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wnqlw"] Jul 24 16:12:10.223685 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.223653 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02cd00be-d34c-4fbf-a373-477a1ed4f46a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wnqlw\" (UID: \"02cd00be-d34c-4fbf-a373-477a1ed4f46a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:10.223856 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.223715 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d95hf\" (UniqueName: \"kubernetes.io/projected/02cd00be-d34c-4fbf-a373-477a1ed4f46a-kube-api-access-d95hf\") pod \"cert-manager-webhook-6486ffd997-wnqlw\" (UID: \"02cd00be-d34c-4fbf-a373-477a1ed4f46a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:10.324576 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.324491 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d95hf\" (UniqueName: \"kubernetes.io/projected/02cd00be-d34c-4fbf-a373-477a1ed4f46a-kube-api-access-d95hf\") pod \"cert-manager-webhook-6486ffd997-wnqlw\" (UID: \"02cd00be-d34c-4fbf-a373-477a1ed4f46a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:10.324576 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.324572 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02cd00be-d34c-4fbf-a373-477a1ed4f46a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wnqlw\" (UID: \"02cd00be-d34c-4fbf-a373-477a1ed4f46a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:10.333485 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.333463 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02cd00be-d34c-4fbf-a373-477a1ed4f46a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wnqlw\" (UID: \"02cd00be-d34c-4fbf-a373-477a1ed4f46a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:10.333852 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.333830 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d95hf\" (UniqueName: \"kubernetes.io/projected/02cd00be-d34c-4fbf-a373-477a1ed4f46a-kube-api-access-d95hf\") pod \"cert-manager-webhook-6486ffd997-wnqlw\" (UID: \"02cd00be-d34c-4fbf-a373-477a1ed4f46a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:10.394137 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.394105 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:10.526368 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.526335 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wnqlw"] Jul 24 16:12:10.530136 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:12:10.530110 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02cd00be_d34c_4fbf_a373_477a1ed4f46a.slice/crio-3b7aef5343bc1c20f11cd25aa4f6458c31d2f246e5ec329945ed1bf3849b1aee WatchSource:0}: Error finding container 3b7aef5343bc1c20f11cd25aa4f6458c31d2f246e5ec329945ed1bf3849b1aee: Status 404 returned error can't find the container with id 3b7aef5343bc1c20f11cd25aa4f6458c31d2f246e5ec329945ed1bf3849b1aee Jul 24 16:12:10.532360 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.532343 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:12:10.777022 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:10.776991 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" event={"ID":"02cd00be-d34c-4fbf-a373-477a1ed4f46a","Type":"ContainerStarted","Data":"3b7aef5343bc1c20f11cd25aa4f6458c31d2f246e5ec329945ed1bf3849b1aee"} Jul 24 16:12:12.063460 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:12.063254 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:12.161734 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:12.161634 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:13.612525 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:13.612480 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:13.792312 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:13.792280 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" event={"ID":"02cd00be-d34c-4fbf-a373-477a1ed4f46a","Type":"ContainerStarted","Data":"e2cc269c8cad4c9c5731e19b4a5a1ad86f7dbb6dedc768fef41d4cf6305b0487"} Jul 24 16:12:13.792444 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:13.792336 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:13.809629 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:13.809582 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" podStartSLOduration=0.693732248 podStartE2EDuration="3.809567598s" totalImagesPullingTime="3.11583535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:12:13.808353004 +0000 UTC m=+353.239988377" watchObservedRunningTime="2026-07-24 16:12:13.809567598 +0000 UTC m=+353.241202958" Jul 24 16:12:14.034619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.034546 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl"] Jul 24 16:12:14.037754 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.037736 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" Jul 24 16:12:14.040520 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.040474 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 16:12:14.040641 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.040628 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-f7cch\"" Jul 24 16:12:14.040692 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.040658 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 16:12:14.047596 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.047578 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl"] Jul 24 16:12:14.161000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.160968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgmm4\" (UniqueName: \"kubernetes.io/projected/c636941c-b9c2-4f0a-98ba-e0bdec7e723a-kube-api-access-wgmm4\") pod \"openshift-lws-operator-fd8ccff4c-hppwl\" (UID: \"c636941c-b9c2-4f0a-98ba-e0bdec7e723a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" Jul 24 16:12:14.161185 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.161029 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c636941c-b9c2-4f0a-98ba-e0bdec7e723a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hppwl\" (UID: \"c636941c-b9c2-4f0a-98ba-e0bdec7e723a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" Jul 24 16:12:14.261766 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.261730 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmm4\" (UniqueName: \"kubernetes.io/projected/c636941c-b9c2-4f0a-98ba-e0bdec7e723a-kube-api-access-wgmm4\") pod \"openshift-lws-operator-fd8ccff4c-hppwl\" (UID: \"c636941c-b9c2-4f0a-98ba-e0bdec7e723a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" Jul 24 16:12:14.261898 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.261781 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c636941c-b9c2-4f0a-98ba-e0bdec7e723a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hppwl\" (UID: \"c636941c-b9c2-4f0a-98ba-e0bdec7e723a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" Jul 24 16:12:14.262191 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.262174 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c636941c-b9c2-4f0a-98ba-e0bdec7e723a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hppwl\" (UID: \"c636941c-b9c2-4f0a-98ba-e0bdec7e723a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" Jul 24 16:12:14.271110 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.271093 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmm4\" (UniqueName: \"kubernetes.io/projected/c636941c-b9c2-4f0a-98ba-e0bdec7e723a-kube-api-access-wgmm4\") pod \"openshift-lws-operator-fd8ccff4c-hppwl\" (UID: \"c636941c-b9c2-4f0a-98ba-e0bdec7e723a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" Jul 24 16:12:14.347106 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.347045 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" Jul 24 16:12:14.477318 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.477296 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl"] Jul 24 16:12:14.479695 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:12:14.479671 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc636941c_b9c2_4f0a_98ba_e0bdec7e723a.slice/crio-f1732de41684b20aad12939c3d4198860080d7f9826210f9fd4659b603a36902 WatchSource:0}: Error finding container f1732de41684b20aad12939c3d4198860080d7f9826210f9fd4659b603a36902: Status 404 returned error can't find the container with id f1732de41684b20aad12939c3d4198860080d7f9826210f9fd4659b603a36902 Jul 24 16:12:14.799804 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:14.799757 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" event={"ID":"c636941c-b9c2-4f0a-98ba-e0bdec7e723a","Type":"ContainerStarted","Data":"f1732de41684b20aad12939c3d4198860080d7f9826210f9fd4659b603a36902"} Jul 24 16:12:16.809788 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:16.809755 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" event={"ID":"c636941c-b9c2-4f0a-98ba-e0bdec7e723a","Type":"ContainerStarted","Data":"e2549d7263250e2a38ce50f6b9fe21f9cefb342f5d9256610168e3de849bec4a"} Jul 24 16:12:16.842763 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:16.842713 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hppwl" podStartSLOduration=0.693310464 podStartE2EDuration="2.8426969s" totalImagesPullingTime="2.149386436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:12:16.841836198 +0000 UTC m=+356.273471558" watchObservedRunningTime="2026-07-24 16:12:16.8426969 +0000 UTC m=+356.274332259" Jul 24 16:12:17.837097 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:17.837067 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:18.029640 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:18.029605 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:19.801587 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:19.801560 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wnqlw" Jul 24 16:12:20.014220 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:20.014184 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:32.184687 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:32.184655 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:34.547758 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:34.547724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:34.642714 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:34.642682 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:35.571885 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:35.571850 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:35.698824 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:35.698792 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:36.372997 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.372964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:36.683733 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.683654 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:36.787134 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.787100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:36.867926 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.867899 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9"] Jul 24 16:12:36.871460 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.871444 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:36.875443 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.875423 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 16:12:36.875655 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.875642 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 16:12:36.875742 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.875718 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zw9nh\"" Jul 24 16:12:36.878430 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.878251 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 16:12:36.878922 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.878906 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 16:12:36.897085 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.897061 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9"] Jul 24 16:12:36.956053 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.955982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdd2b47f-636d-4e3f-b397-92a44e4268b7-manifests\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:36.956053 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.956014 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdd2b47f-636d-4e3f-b397-92a44e4268b7-tmp\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:36.956262 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.956117 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdd2b47f-636d-4e3f-b397-92a44e4268b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:36.956262 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.956168 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwlv6\" (UniqueName: \"kubernetes.io/projected/bdd2b47f-636d-4e3f-b397-92a44e4268b7-kube-api-access-xwlv6\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:36.956262 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:36.956235 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdd2b47f-636d-4e3f-b397-92a44e4268b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.025646 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.025613 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:37.056818 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.056793 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdd2b47f-636d-4e3f-b397-92a44e4268b7-manifests\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.056933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.056821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdd2b47f-636d-4e3f-b397-92a44e4268b7-tmp\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.056933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.056871 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdd2b47f-636d-4e3f-b397-92a44e4268b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.056933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.056888 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwlv6\" (UniqueName: \"kubernetes.io/projected/bdd2b47f-636d-4e3f-b397-92a44e4268b7-kube-api-access-xwlv6\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.056933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.056919 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdd2b47f-636d-4e3f-b397-92a44e4268b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.057249 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.057228 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdd2b47f-636d-4e3f-b397-92a44e4268b7-manifests\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.059373 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.059349 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdd2b47f-636d-4e3f-b397-92a44e4268b7-tmp\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.059493 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.059477 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdd2b47f-636d-4e3f-b397-92a44e4268b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.059546 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.059525 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdd2b47f-636d-4e3f-b397-92a44e4268b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.085710 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.085684 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwlv6\" (UniqueName: \"kubernetes.io/projected/bdd2b47f-636d-4e3f-b397-92a44e4268b7-kube-api-access-xwlv6\") pod \"opendatahub-operator-controller-manager-5d6c795b6-5t7s9\" (UID: \"bdd2b47f-636d-4e3f-b397-92a44e4268b7\") " pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.181710 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.181681 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:37.210450 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.210394 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:37.302538 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.302501 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:37.326546 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.326523 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9"] Jul 24 16:12:37.328625 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:12:37.328595 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd2b47f_636d_4e3f_b397_92a44e4268b7.slice/crio-b32b9aba5e5eec43b7490543ac70ce99d4f5d4346822c263e32085058dfbcf39 WatchSource:0}: Error finding container b32b9aba5e5eec43b7490543ac70ce99d4f5d4346822c263e32085058dfbcf39: Status 404 returned error can't find the container with id b32b9aba5e5eec43b7490543ac70ce99d4f5d4346822c263e32085058dfbcf39 Jul 24 16:12:37.409540 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.409511 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:37.515062 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.514993 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:37.623570 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.623525 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:37.728822 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.728764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:37.900238 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:37.900205 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" event={"ID":"bdd2b47f-636d-4e3f-b397-92a44e4268b7","Type":"ContainerStarted","Data":"b32b9aba5e5eec43b7490543ac70ce99d4f5d4346822c263e32085058dfbcf39"} Jul 24 16:12:40.919099 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:40.919065 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" event={"ID":"bdd2b47f-636d-4e3f-b397-92a44e4268b7","Type":"ContainerStarted","Data":"96befad403d7a742e6bc50bf0f57b092b0f3307bd01b514a600a3e6574ea7987"} Jul 24 16:12:41.924900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:41.924864 2560 generic.go:356] "Generic (PLEG): container finished" podID="bdd2b47f-636d-4e3f-b397-92a44e4268b7" containerID="96befad403d7a742e6bc50bf0f57b092b0f3307bd01b514a600a3e6574ea7987" exitCode=0 Jul 24 16:12:41.925287 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:41.924907 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" event={"ID":"bdd2b47f-636d-4e3f-b397-92a44e4268b7","Type":"ContainerDied","Data":"96befad403d7a742e6bc50bf0f57b092b0f3307bd01b514a600a3e6574ea7987"} Jul 24 16:12:42.930338 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:42.930303 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" event={"ID":"bdd2b47f-636d-4e3f-b397-92a44e4268b7","Type":"ContainerStarted","Data":"fe31d18f6344a344e8d4bbcb4376c777946125afaef8af40570ac35215195d45"} Jul 24 16:12:42.930778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:42.930438 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:42.958207 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:42.958161 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" podStartSLOduration=2.353643443 podStartE2EDuration="6.958128454s" totalImagesPullingTime="3.595965798s" totalInitContainerRuntime="1.008519213s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:12:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:12:42.955464368 +0000 UTC m=+382.387099727" watchObservedRunningTime="2026-07-24 16:12:42.958128454 +0000 UTC m=+382.389763813" Jul 24 16:12:52.890286 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:52.890251 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:52.981555 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:52.981522 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:53.936944 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:53.936916 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d6c795b6-5t7s9" Jul 24 16:12:54.877937 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:54.877901 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj"] Jul 24 16:12:54.885875 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:54.885850 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:54.889422 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:54.889391 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 16:12:54.889422 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:54.889405 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cxf24\"" Jul 24 16:12:54.889613 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:54.889412 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 16:12:54.889613 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:54.889447 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 16:12:54.896039 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:54.896016 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj"] Jul 24 16:12:55.012825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.012795 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-cert\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.013229 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.012839 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65b5m\" (UniqueName: \"kubernetes.io/projected/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-kube-api-access-65b5m\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.013229 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.012946 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-manager-config\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.013229 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.012985 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-metrics-cert\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.114236 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.114204 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-metrics-cert\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.114398 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.114247 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-cert\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.114398 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.114372 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65b5m\" (UniqueName: \"kubernetes.io/projected/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-kube-api-access-65b5m\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.114512 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.114497 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-manager-config\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.115169 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.115122 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-manager-config\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.116970 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.116949 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-metrics-cert\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.116970 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.116959 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-cert\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.123570 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.123548 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65b5m\" (UniqueName: \"kubernetes.io/projected/b104b9a1-fb54-4f03-a35e-53b6d37a8ff6-kube-api-access-65b5m\") pod \"lws-controller-manager-9cffb5c7d-45vrj\" (UID: \"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6\") " pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.195427 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.195358 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:55.326761 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.326740 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj"] Jul 24 16:12:55.328974 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:12:55.328949 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb104b9a1_fb54_4f03_a35e_53b6d37a8ff6.slice/crio-30c4c36291e2708950d0a91bc8ee6833b8ede3130e980105ebb5d7ed4e2b5dde WatchSource:0}: Error finding container 30c4c36291e2708950d0a91bc8ee6833b8ede3130e980105ebb5d7ed4e2b5dde: Status 404 returned error can't find the container with id 30c4c36291e2708950d0a91bc8ee6833b8ede3130e980105ebb5d7ed4e2b5dde Jul 24 16:12:55.990189 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:55.990137 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" event={"ID":"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6","Type":"ContainerStarted","Data":"30c4c36291e2708950d0a91bc8ee6833b8ede3130e980105ebb5d7ed4e2b5dde"} Jul 24 16:12:56.639430 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:56.639395 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:56.823952 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:56.823918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:57.282301 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.282216 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:57.699256 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.699221 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh"] Jul 24 16:12:57.701627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.701603 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:57.705323 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.705258 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 16:12:57.705699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.705493 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 16:12:57.705814 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.705767 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 16:12:57.706322 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.706167 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 16:12:57.709356 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.709182 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sqhqs\"" Jul 24 16:12:57.721761 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.721740 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh"] Jul 24 16:12:57.839869 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.839839 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fc0c646-2f90-4fc0-b497-6322d0f84a12-tmp\") pod \"kube-auth-proxy-cd59bb69-wqxgh\" (UID: \"6fc0c646-2f90-4fc0-b497-6322d0f84a12\") " pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:57.840031 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.839879 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc0c646-2f90-4fc0-b497-6322d0f84a12-tls-certs\") pod \"kube-auth-proxy-cd59bb69-wqxgh\" (UID: \"6fc0c646-2f90-4fc0-b497-6322d0f84a12\") " pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:57.840031 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.839902 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mjc\" (UniqueName: \"kubernetes.io/projected/6fc0c646-2f90-4fc0-b497-6322d0f84a12-kube-api-access-t5mjc\") pod \"kube-auth-proxy-cd59bb69-wqxgh\" (UID: \"6fc0c646-2f90-4fc0-b497-6322d0f84a12\") " pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:57.941121 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.941088 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc0c646-2f90-4fc0-b497-6322d0f84a12-tls-certs\") pod \"kube-auth-proxy-cd59bb69-wqxgh\" (UID: \"6fc0c646-2f90-4fc0-b497-6322d0f84a12\") " pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:57.941121 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.941127 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mjc\" (UniqueName: \"kubernetes.io/projected/6fc0c646-2f90-4fc0-b497-6322d0f84a12-kube-api-access-t5mjc\") pod \"kube-auth-proxy-cd59bb69-wqxgh\" (UID: \"6fc0c646-2f90-4fc0-b497-6322d0f84a12\") " pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:57.941395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.941218 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fc0c646-2f90-4fc0-b497-6322d0f84a12-tmp\") pod \"kube-auth-proxy-cd59bb69-wqxgh\" (UID: \"6fc0c646-2f90-4fc0-b497-6322d0f84a12\") " pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:57.943734 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.943709 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fc0c646-2f90-4fc0-b497-6322d0f84a12-tmp\") pod \"kube-auth-proxy-cd59bb69-wqxgh\" (UID: \"6fc0c646-2f90-4fc0-b497-6322d0f84a12\") " pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:57.943859 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.943759 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc0c646-2f90-4fc0-b497-6322d0f84a12-tls-certs\") pod \"kube-auth-proxy-cd59bb69-wqxgh\" (UID: \"6fc0c646-2f90-4fc0-b497-6322d0f84a12\") " pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:57.953666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.953579 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:57.955073 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:57.955052 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mjc\" (UniqueName: \"kubernetes.io/projected/6fc0c646-2f90-4fc0-b497-6322d0f84a12-kube-api-access-t5mjc\") pod \"kube-auth-proxy-cd59bb69-wqxgh\" (UID: \"6fc0c646-2f90-4fc0-b497-6322d0f84a12\") " pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:58.002063 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:58.002036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" event={"ID":"b104b9a1-fb54-4f03-a35e-53b6d37a8ff6","Type":"ContainerStarted","Data":"4c9cf25968b7533bae40387b997ca4b3a2617e85789ef3d738841382344ffa3f"} Jul 24 16:12:58.002235 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:58.002095 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:12:58.015949 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:58.015918 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" Jul 24 16:12:58.027267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:58.027222 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" podStartSLOduration=2.32838668 podStartE2EDuration="4.027201939s" totalImagesPullingTime="1.698815259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:12:58.026229241 +0000 UTC m=+397.457864601" watchObservedRunningTime="2026-07-24 16:12:58.027201939 +0000 UTC m=+397.458837300" Jul 24 16:12:58.051736 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:58.051707 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:58.156707 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:58.156680 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh"] Jul 24 16:12:58.159539 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:58.159512 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:58.160842 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:12:58.160814 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc0c646_2f90_4fc0_b497_6322d0f84a12.slice/crio-bcaabae4ed63df5fa4cd698f9fe933eca00ee80fb3437ac69b78e5d017f50359 WatchSource:0}: Error finding container bcaabae4ed63df5fa4cd698f9fe933eca00ee80fb3437ac69b78e5d017f50359: Status 404 returned error can't find the container with id bcaabae4ed63df5fa4cd698f9fe933eca00ee80fb3437ac69b78e5d017f50359 Jul 24 16:12:58.251931 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:58.251845 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:58.354491 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:58.354448 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:59.010399 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:59.010364 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" event={"ID":"6fc0c646-2f90-4fc0-b497-6322d0f84a12","Type":"ContainerStarted","Data":"bcaabae4ed63df5fa4cd698f9fe933eca00ee80fb3437ac69b78e5d017f50359"} Jul 24 16:12:59.173095 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:59.173066 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:59.310775 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:59.310688 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:59.432654 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:59.432622 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:12:59.619909 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:12:59.619876 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:00.800676 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:00.800644 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:01.021554 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:01.021515 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" event={"ID":"6fc0c646-2f90-4fc0-b497-6322d0f84a12","Type":"ContainerStarted","Data":"2eb8fcda3d56c33340391d7f1249925c464382552481524fec077f5c1b0860ec"} Jul 24 16:13:01.044209 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:01.044131 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cd59bb69-wqxgh" podStartSLOduration=1.322205867 podStartE2EDuration="4.044116694s" totalImagesPullingTime="2.721910827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:13:01.04352553 +0000 UTC m=+400.475160890" watchObservedRunningTime="2026-07-24 16:13:01.044116694 +0000 UTC m=+400.475752100" Jul 24 16:13:05.016329 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:05.016293 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:05.120279 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:05.120240 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:05.962819 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:05.962783 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:06.207545 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:06.207491 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:06.331226 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:06.331134 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:09.013470 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:09.013438 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9cffb5c7d-45vrj" Jul 24 16:13:09.188089 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:09.188058 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:09.383760 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:09.383731 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:09.504935 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:09.504899 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:09.720313 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:09.720221 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:09.988599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:09.988516 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:10.155721 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:10.155690 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:16.006342 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:16.006306 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:16.083528 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:16.083499 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:18.405962 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:18.405922 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:18.492262 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:18.492231 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:58.155048 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:58.154967 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:58.258230 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:58.258200 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:58.342312 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:58.342276 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:59.079011 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:59.078985 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:13:59.262104 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:13:59.262069 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:27.476521 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.476491 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cdc4c448b-6srp6"] Jul 24 16:14:27.478712 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.478696 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.497101 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.497065 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdc4c448b-6srp6"] Jul 24 16:14:27.508644 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.508622 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71ef4113-c810-4121-bbdc-22dffe8a3fc2-console-serving-cert\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.508762 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.508656 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjdsw\" (UniqueName: \"kubernetes.io/projected/71ef4113-c810-4121-bbdc-22dffe8a3fc2-kube-api-access-cjdsw\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.508762 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.508739 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71ef4113-c810-4121-bbdc-22dffe8a3fc2-console-oauth-config\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.508881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.508796 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-service-ca\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.508881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.508821 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-oauth-serving-cert\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.508963 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.508888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-trusted-ca-bundle\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.508963 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.508936 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-console-config\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.572395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.572373 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:27.610282 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.610255 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-trusted-ca-bundle\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.610395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.610289 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-console-config\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.610395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.610339 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71ef4113-c810-4121-bbdc-22dffe8a3fc2-console-serving-cert\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.610395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.610368 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjdsw\" (UniqueName: \"kubernetes.io/projected/71ef4113-c810-4121-bbdc-22dffe8a3fc2-kube-api-access-cjdsw\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.610562 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.610427 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71ef4113-c810-4121-bbdc-22dffe8a3fc2-console-oauth-config\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.610562 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.610480 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-service-ca\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.610562 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.610505 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-oauth-serving-cert\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.611120 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.611094 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-console-config\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.611423 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.611128 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-service-ca\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.611423 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.611184 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-trusted-ca-bundle\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.611423 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.611390 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71ef4113-c810-4121-bbdc-22dffe8a3fc2-oauth-serving-cert\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.612908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.612885 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71ef4113-c810-4121-bbdc-22dffe8a3fc2-console-serving-cert\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.612985 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.612948 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71ef4113-c810-4121-bbdc-22dffe8a3fc2-console-oauth-config\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.626285 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.626266 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjdsw\" (UniqueName: \"kubernetes.io/projected/71ef4113-c810-4121-bbdc-22dffe8a3fc2-kube-api-access-cjdsw\") pod \"console-7cdc4c448b-6srp6\" (UID: \"71ef4113-c810-4121-bbdc-22dffe8a3fc2\") " pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.677930 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.677904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:27.787710 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.787660 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:27.921406 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:27.921384 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdc4c448b-6srp6"] Jul 24 16:14:27.923731 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:14:27.923705 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ef4113_c810_4121_bbdc_22dffe8a3fc2.slice/crio-604fef253893d8871ac6fd479172045a1b2947e8731743ec21fce8ae057b3df4 WatchSource:0}: Error finding container 604fef253893d8871ac6fd479172045a1b2947e8731743ec21fce8ae057b3df4: Status 404 returned error can't find the container with id 604fef253893d8871ac6fd479172045a1b2947e8731743ec21fce8ae057b3df4 Jul 24 16:14:28.074603 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:28.074528 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:28.412947 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:28.412910 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdc4c448b-6srp6" event={"ID":"71ef4113-c810-4121-bbdc-22dffe8a3fc2","Type":"ContainerStarted","Data":"c2ba9ffa1b24a5e032bc1b2ad74893a77a629a00e8a87cda6cd2f89613265849"} Jul 24 16:14:28.413228 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:28.413010 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdc4c448b-6srp6" event={"ID":"71ef4113-c810-4121-bbdc-22dffe8a3fc2","Type":"ContainerStarted","Data":"604fef253893d8871ac6fd479172045a1b2947e8731743ec21fce8ae057b3df4"} Jul 24 16:14:28.436713 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:28.436670 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cdc4c448b-6srp6" podStartSLOduration=1.436655119 podStartE2EDuration="1.436655119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:14:28.43483767 +0000 UTC m=+487.866473031" watchObservedRunningTime="2026-07-24 16:14:28.436655119 +0000 UTC m=+487.868290478" Jul 24 16:14:28.909591 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:28.909559 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:33.285269 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:33.285219 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:35.193680 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:35.193645 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:35.338795 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:35.338755 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:37.739569 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:37.739536 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:37.788316 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:37.788288 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:37.788465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:37.788343 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:37.792795 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:37.792774 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:38.463211 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:38.463185 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cdc4c448b-6srp6" Jul 24 16:14:38.476290 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:38.476260 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:38.544694 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:38.544658 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f95b8b676-pp5n9"] Jul 24 16:14:38.711556 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:38.711524 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:38.838972 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:38.838870 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:38.940775 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:38.940726 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:41.067205 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:41.067173 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:41.348358 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:41.348260 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:44.507611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:44.507580 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:46.056742 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.056711 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:46.158272 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.158240 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:46.340063 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.339982 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp"] Jul 24 16:14:46.343066 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.343045 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:46.346000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.345974 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 16:14:46.346113 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.346037 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 16:14:46.346113 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.346038 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 16:14:46.346684 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.346669 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 16:14:46.346684 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.346677 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w6clx\"" Jul 24 16:14:46.353943 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.353923 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp"] Jul 24 16:14:46.359052 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.359033 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:46.486726 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.486698 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/38632bce-1983-4157-87d4-e2c99662e222-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:46.486889 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.486773 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99b7q\" (UniqueName: \"kubernetes.io/projected/38632bce-1983-4157-87d4-e2c99662e222-kube-api-access-99b7q\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:46.486889 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.486799 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38632bce-1983-4157-87d4-e2c99662e222-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:46.587543 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.587518 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99b7q\" (UniqueName: \"kubernetes.io/projected/38632bce-1983-4157-87d4-e2c99662e222-kube-api-access-99b7q\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:46.587701 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.587553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38632bce-1983-4157-87d4-e2c99662e222-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:46.587701 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.587595 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/38632bce-1983-4157-87d4-e2c99662e222-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:46.587789 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:14:46.587706 2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 24 16:14:46.587789 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:14:46.587760 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38632bce-1983-4157-87d4-e2c99662e222-plugin-serving-cert podName:38632bce-1983-4157-87d4-e2c99662e222 nodeName:}" failed. No retries permitted until 2026-07-24 16:14:47.087745872 +0000 UTC m=+506.519381210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/38632bce-1983-4157-87d4-e2c99662e222-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-w59lp" (UID: "38632bce-1983-4157-87d4-e2c99662e222") : secret "plugin-serving-cert" not found Jul 24 16:14:46.588290 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.588273 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38632bce-1983-4157-87d4-e2c99662e222-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:46.598172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.598112 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99b7q\" (UniqueName: \"kubernetes.io/projected/38632bce-1983-4157-87d4-e2c99662e222-kube-api-access-99b7q\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:46.688396 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.688370 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:46.794848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:46.794819 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:47.091619 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:47.091582 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/38632bce-1983-4157-87d4-e2c99662e222-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:47.094124 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:47.094106 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/38632bce-1983-4157-87d4-e2c99662e222-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w59lp\" (UID: \"38632bce-1983-4157-87d4-e2c99662e222\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:47.253224 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:47.253193 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" Jul 24 16:14:47.403040 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:47.403014 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp"] Jul 24 16:14:47.404976 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:14:47.404953 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38632bce_1983_4157_87d4_e2c99662e222.slice/crio-b8c52d2a22445b7b50bf4c80d6f167fd1579ebf7267504d27364f50e5841b9f4 WatchSource:0}: Error finding container b8c52d2a22445b7b50bf4c80d6f167fd1579ebf7267504d27364f50e5841b9f4: Status 404 returned error can't find the container with id b8c52d2a22445b7b50bf4c80d6f167fd1579ebf7267504d27364f50e5841b9f4 Jul 24 16:14:47.501017 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:47.500982 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" event={"ID":"38632bce-1983-4157-87d4-e2c99662e222","Type":"ContainerStarted","Data":"b8c52d2a22445b7b50bf4c80d6f167fd1579ebf7267504d27364f50e5841b9f4"} Jul 24 16:14:51.067351 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:51.067306 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:51.168680 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:51.168616 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:59.289597 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:59.289527 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:14:59.396054 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:14:59.396001 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:03.461090 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:03.461050 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:03.560689 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:03.560655 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:03.564580 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:03.564526 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f95b8b676-pp5n9" podUID="59813405-ba3f-4f3f-832c-17d807ee62d1" containerName="console" containerID="cri-o://dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81" gracePeriod=15 Jul 24 16:15:07.460287 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:07.460252 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:07.561280 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:07.561250 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:08.046327 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:08.046296 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:08.123112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:08.123076 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:09.546712 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.546692 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:15:09.612833 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.612799 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" event={"ID":"38632bce-1983-4157-87d4-e2c99662e222","Type":"ContainerStarted","Data":"786e5974fc43b90a425812fe13e4db1dc59cb46beb89e4628cb5b01a39139895"} Jul 24 16:15:09.614102 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.614078 2560 generic.go:356] "Generic (PLEG): container finished" podID="59813405-ba3f-4f3f-832c-17d807ee62d1" containerID="dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81" exitCode=0 Jul 24 16:15:09.614232 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.614116 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f95b8b676-pp5n9" Jul 24 16:15:09.614232 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.614187 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f95b8b676-pp5n9" event={"ID":"59813405-ba3f-4f3f-832c-17d807ee62d1","Type":"ContainerDied","Data":"dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81"} Jul 24 16:15:09.614312 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.614253 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f95b8b676-pp5n9" event={"ID":"59813405-ba3f-4f3f-832c-17d807ee62d1","Type":"ContainerDied","Data":"11dd3855011f1005d3a52188f88375b9f126ef4cd1a1396e78c336070eafc69b"} Jul 24 16:15:09.614312 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.614274 2560 scope.go:117] "RemoveContainer" containerID="dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81" Jul 24 16:15:09.622581 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.622562 2560 scope.go:117] "RemoveContainer" containerID="dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81" Jul 24 16:15:09.622846 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:15:09.622828 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81\": container with ID starting with dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81 not found: ID does not exist" containerID="dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81" Jul 24 16:15:09.622893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.622853 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81"} err="failed to get container status \"dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81\": rpc error: code = NotFound desc = could not find container \"dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81\": container with ID starting with dbd5add97118321735af862f0e44c11dba2f004dac65c948a654b6fc4edf0b81 not found: ID does not exist" Jul 24 16:15:09.628091 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628076 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29dlr\" (UniqueName: \"kubernetes.io/projected/59813405-ba3f-4f3f-832c-17d807ee62d1-kube-api-access-29dlr\") pod \"59813405-ba3f-4f3f-832c-17d807ee62d1\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " Jul 24 16:15:09.628177 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628129 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-oauth-serving-cert\") pod \"59813405-ba3f-4f3f-832c-17d807ee62d1\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " Jul 24 16:15:09.628238 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628179 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-service-ca\") pod \"59813405-ba3f-4f3f-832c-17d807ee62d1\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " Jul 24 16:15:09.628298 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628248 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-trusted-ca-bundle\") pod \"59813405-ba3f-4f3f-832c-17d807ee62d1\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " Jul 24 16:15:09.628356 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628317 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-serving-cert\") pod \"59813405-ba3f-4f3f-832c-17d807ee62d1\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " Jul 24 16:15:09.628408 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628370 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-console-config\") pod \"59813405-ba3f-4f3f-832c-17d807ee62d1\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " Jul 24 16:15:09.628408 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628395 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-oauth-config\") pod \"59813405-ba3f-4f3f-832c-17d807ee62d1\" (UID: \"59813405-ba3f-4f3f-832c-17d807ee62d1\") " Jul 24 16:15:09.628691 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628647 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "59813405-ba3f-4f3f-832c-17d807ee62d1" (UID: "59813405-ba3f-4f3f-832c-17d807ee62d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:15:09.628840 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628581 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-service-ca" (OuterVolumeSpecName: "service-ca") pod "59813405-ba3f-4f3f-832c-17d807ee62d1" (UID: "59813405-ba3f-4f3f-832c-17d807ee62d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:15:09.628907 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628689 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "59813405-ba3f-4f3f-832c-17d807ee62d1" (UID: "59813405-ba3f-4f3f-832c-17d807ee62d1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:15:09.628907 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.628771 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-console-config" (OuterVolumeSpecName: "console-config") pod "59813405-ba3f-4f3f-832c-17d807ee62d1" (UID: "59813405-ba3f-4f3f-832c-17d807ee62d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:15:09.630592 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.630548 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59813405-ba3f-4f3f-832c-17d807ee62d1-kube-api-access-29dlr" (OuterVolumeSpecName: "kube-api-access-29dlr") pod "59813405-ba3f-4f3f-832c-17d807ee62d1" (UID: "59813405-ba3f-4f3f-832c-17d807ee62d1"). InnerVolumeSpecName "kube-api-access-29dlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:15:09.630660 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.630610 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "59813405-ba3f-4f3f-832c-17d807ee62d1" (UID: "59813405-ba3f-4f3f-832c-17d807ee62d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:15:09.630864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.630848 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "59813405-ba3f-4f3f-832c-17d807ee62d1" (UID: "59813405-ba3f-4f3f-832c-17d807ee62d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:15:09.637287 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.637247 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w59lp" podStartSLOduration=1.573221306 podStartE2EDuration="23.63722647s" totalImagesPullingTime="22.064005164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:15:09.634432934 +0000 UTC m=+529.066068293" watchObservedRunningTime="2026-07-24 16:15:09.63722647 +0000 UTC m=+529.068861829" Jul 24 16:15:09.729976 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.729947 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-trusted-ca-bundle\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:15:09.729976 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.729972 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:15:09.729976 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.729983 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-console-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:15:09.730243 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.729992 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59813405-ba3f-4f3f-832c-17d807ee62d1-console-oauth-config\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:15:09.730243 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.730004 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29dlr\" (UniqueName: \"kubernetes.io/projected/59813405-ba3f-4f3f-832c-17d807ee62d1-kube-api-access-29dlr\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:15:09.730243 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.730017 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-oauth-serving-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:15:09.730243 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.730029 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59813405-ba3f-4f3f-832c-17d807ee62d1-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:15:09.958291 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.958258 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f95b8b676-pp5n9"] Jul 24 16:15:09.962620 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:09.962596 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f95b8b676-pp5n9"] Jul 24 16:15:11.167166 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:11.167073 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59813405-ba3f-4f3f-832c-17d807ee62d1" path="/var/lib/kubelet/pods/59813405-ba3f-4f3f-832c-17d807ee62d1/volumes" Jul 24 16:15:11.960293 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:11.960263 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:12.066413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:12.066386 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:12.207928 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:12.207900 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:12.312621 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:12.312532 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:12.633821 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:12.633794 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:12.734780 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:12.734747 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:17.768589 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:17.768508 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:17.891194 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:17.891160 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:22.516101 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:22.516060 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:22.618897 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:22.618861 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:25.456140 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:25.456106 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:25.743004 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:25.742923 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:25.838170 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:25.838116 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:25.950890 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:25.950860 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:26.321927 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:26.321899 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:26.423620 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:26.423595 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:28.366102 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.366064 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:28.404161 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.404117 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cpn6g"] Jul 24 16:15:28.404536 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.404522 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59813405-ba3f-4f3f-832c-17d807ee62d1" containerName="console" Jul 24 16:15:28.404578 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.404541 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="59813405-ba3f-4f3f-832c-17d807ee62d1" containerName="console" Jul 24 16:15:28.404618 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.404599 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="59813405-ba3f-4f3f-832c-17d807ee62d1" containerName="console" Jul 24 16:15:28.468472 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.468439 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:28.484409 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.484385 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cpn6g"] Jul 24 16:15:28.484409 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.484416 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cpn6g"] Jul 24 16:15:28.484580 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.484492 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:28.488011 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.487990 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 16:15:28.564627 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.564599 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:28.590972 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.590951 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/908feb44-5edc-4596-a8e2-5a771fc908aa-config-file\") pod \"limitador-limitador-64c94fd64f-cpn6g\" (UID: \"908feb44-5edc-4596-a8e2-5a771fc908aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:28.591110 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.590976 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrv2g\" (UniqueName: \"kubernetes.io/projected/908feb44-5edc-4596-a8e2-5a771fc908aa-kube-api-access-hrv2g\") pod \"limitador-limitador-64c94fd64f-cpn6g\" (UID: \"908feb44-5edc-4596-a8e2-5a771fc908aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:28.692039 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.691960 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/908feb44-5edc-4596-a8e2-5a771fc908aa-config-file\") pod \"limitador-limitador-64c94fd64f-cpn6g\" (UID: \"908feb44-5edc-4596-a8e2-5a771fc908aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:28.692039 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.691994 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrv2g\" (UniqueName: \"kubernetes.io/projected/908feb44-5edc-4596-a8e2-5a771fc908aa-kube-api-access-hrv2g\") pod \"limitador-limitador-64c94fd64f-cpn6g\" (UID: \"908feb44-5edc-4596-a8e2-5a771fc908aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:28.692568 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.692546 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/908feb44-5edc-4596-a8e2-5a771fc908aa-config-file\") pod \"limitador-limitador-64c94fd64f-cpn6g\" (UID: \"908feb44-5edc-4596-a8e2-5a771fc908aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:28.702211 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.702189 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrv2g\" (UniqueName: \"kubernetes.io/projected/908feb44-5edc-4596-a8e2-5a771fc908aa-kube-api-access-hrv2g\") pod \"limitador-limitador-64c94fd64f-cpn6g\" (UID: \"908feb44-5edc-4596-a8e2-5a771fc908aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:28.794162 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.794107 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:28.934671 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:28.934647 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cpn6g"] Jul 24 16:15:28.937061 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:15:28.937034 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908feb44_5edc_4596_a8e2_5a771fc908aa.slice/crio-4eb87d0976855f731eeaf0d3f49ae66d53562f2c761520c4b2ba964ac8c385d0 WatchSource:0}: Error finding container 4eb87d0976855f731eeaf0d3f49ae66d53562f2c761520c4b2ba964ac8c385d0: Status 404 returned error can't find the container with id 4eb87d0976855f731eeaf0d3f49ae66d53562f2c761520c4b2ba964ac8c385d0 Jul 24 16:15:29.263712 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.263678 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:29.348830 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.348793 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:29.371264 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.371234 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-wp95z"] Jul 24 16:15:29.375648 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.375632 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wp95z" Jul 24 16:15:29.378513 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.378487 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w5pns\"" Jul 24 16:15:29.382260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.382237 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wp95z"] Jul 24 16:15:29.398533 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.398493 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9kp\" (UniqueName: \"kubernetes.io/projected/b62da17d-1c52-40ad-b3b5-03439e120ab2-kube-api-access-6z9kp\") pod \"authorino-75d556bd77-wp95z\" (UID: \"b62da17d-1c52-40ad-b3b5-03439e120ab2\") " pod="kuadrant-system/authorino-75d556bd77-wp95z" Jul 24 16:15:29.498987 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.498955 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9kp\" (UniqueName: \"kubernetes.io/projected/b62da17d-1c52-40ad-b3b5-03439e120ab2-kube-api-access-6z9kp\") pod \"authorino-75d556bd77-wp95z\" (UID: \"b62da17d-1c52-40ad-b3b5-03439e120ab2\") " pod="kuadrant-system/authorino-75d556bd77-wp95z" Jul 24 16:15:29.508303 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.508280 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9kp\" (UniqueName: \"kubernetes.io/projected/b62da17d-1c52-40ad-b3b5-03439e120ab2-kube-api-access-6z9kp\") pod \"authorino-75d556bd77-wp95z\" (UID: \"b62da17d-1c52-40ad-b3b5-03439e120ab2\") " pod="kuadrant-system/authorino-75d556bd77-wp95z" Jul 24 16:15:29.686117 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.686083 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wp95z" Jul 24 16:15:29.710215 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.710182 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" event={"ID":"908feb44-5edc-4596-a8e2-5a771fc908aa","Type":"ContainerStarted","Data":"4eb87d0976855f731eeaf0d3f49ae66d53562f2c761520c4b2ba964ac8c385d0"} Jul 24 16:15:29.860876 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:29.860179 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wp95z"] Jul 24 16:15:29.863058 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:15:29.862935 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb62da17d_1c52_40ad_b3b5_03439e120ab2.slice/crio-b0ae637287c33f0f8e9224738053ad89f5bce3ad3518bc01de26b82ea3c2f254 WatchSource:0}: Error finding container b0ae637287c33f0f8e9224738053ad89f5bce3ad3518bc01de26b82ea3c2f254: Status 404 returned error can't find the container with id b0ae637287c33f0f8e9224738053ad89f5bce3ad3518bc01de26b82ea3c2f254 Jul 24 16:15:30.495241 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:30.495139 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:30.570438 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:30.570402 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:30.721047 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:30.720914 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wp95z" event={"ID":"b62da17d-1c52-40ad-b3b5-03439e120ab2","Type":"ContainerStarted","Data":"b0ae637287c33f0f8e9224738053ad89f5bce3ad3518bc01de26b82ea3c2f254"} Jul 24 16:15:33.740512 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:33.740479 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wp95z" event={"ID":"b62da17d-1c52-40ad-b3b5-03439e120ab2","Type":"ContainerStarted","Data":"0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342"} Jul 24 16:15:33.742458 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:33.742396 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" event={"ID":"908feb44-5edc-4596-a8e2-5a771fc908aa","Type":"ContainerStarted","Data":"74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277"} Jul 24 16:15:33.742643 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:33.742532 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:33.763580 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:33.763531 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-wp95z" podStartSLOduration=1.028956008 podStartE2EDuration="4.763517144s" totalImagesPullingTime="3.734561136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:15:33.761868989 +0000 UTC m=+553.193504349" watchObservedRunningTime="2026-07-24 16:15:33.763517144 +0000 UTC m=+553.195152504" Jul 24 16:15:33.792588 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:33.792539 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" podStartSLOduration=1.13400098 podStartE2EDuration="5.792519994s" totalImagesPullingTime="4.658519014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:15:33.792021349 +0000 UTC m=+553.223656708" watchObservedRunningTime="2026-07-24 16:15:33.792519994 +0000 UTC m=+553.224155360" Jul 24 16:15:38.119060 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:38.119028 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:38.202784 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:38.202755 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:41.894342 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:41.894313 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:41.995260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:41.995231 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:43.736832 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:43.736795 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cpn6g"] Jul 24 16:15:43.737302 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:43.737088 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" podUID="908feb44-5edc-4596-a8e2-5a771fc908aa" containerName="limitador" containerID="cri-o://74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277" gracePeriod=30 Jul 24 16:15:43.737872 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:43.737759 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:44.283629 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.283601 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:44.336663 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.336568 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrv2g\" (UniqueName: \"kubernetes.io/projected/908feb44-5edc-4596-a8e2-5a771fc908aa-kube-api-access-hrv2g\") pod \"908feb44-5edc-4596-a8e2-5a771fc908aa\" (UID: \"908feb44-5edc-4596-a8e2-5a771fc908aa\") " Jul 24 16:15:44.336663 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.336622 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/908feb44-5edc-4596-a8e2-5a771fc908aa-config-file\") pod \"908feb44-5edc-4596-a8e2-5a771fc908aa\" (UID: \"908feb44-5edc-4596-a8e2-5a771fc908aa\") " Jul 24 16:15:44.337017 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.336993 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908feb44-5edc-4596-a8e2-5a771fc908aa-config-file" (OuterVolumeSpecName: "config-file") pod "908feb44-5edc-4596-a8e2-5a771fc908aa" (UID: "908feb44-5edc-4596-a8e2-5a771fc908aa"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:15:44.338971 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.338952 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908feb44-5edc-4596-a8e2-5a771fc908aa-kube-api-access-hrv2g" (OuterVolumeSpecName: "kube-api-access-hrv2g") pod "908feb44-5edc-4596-a8e2-5a771fc908aa" (UID: "908feb44-5edc-4596-a8e2-5a771fc908aa"). InnerVolumeSpecName "kube-api-access-hrv2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:15:44.438085 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.438037 2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/908feb44-5edc-4596-a8e2-5a771fc908aa-config-file\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:15:44.438085 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.438079 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrv2g\" (UniqueName: \"kubernetes.io/projected/908feb44-5edc-4596-a8e2-5a771fc908aa-kube-api-access-hrv2g\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:15:44.798742 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.798709 2560 generic.go:356] "Generic (PLEG): container finished" podID="908feb44-5edc-4596-a8e2-5a771fc908aa" containerID="74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277" exitCode=0 Jul 24 16:15:44.799236 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.798755 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" Jul 24 16:15:44.799236 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.798762 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" event={"ID":"908feb44-5edc-4596-a8e2-5a771fc908aa","Type":"ContainerDied","Data":"74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277"} Jul 24 16:15:44.799236 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.798803 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cpn6g" event={"ID":"908feb44-5edc-4596-a8e2-5a771fc908aa","Type":"ContainerDied","Data":"4eb87d0976855f731eeaf0d3f49ae66d53562f2c761520c4b2ba964ac8c385d0"} Jul 24 16:15:44.799236 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.798824 2560 scope.go:117] "RemoveContainer" containerID="74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277" Jul 24 16:15:44.807271 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.807255 2560 scope.go:117] "RemoveContainer" containerID="74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277" Jul 24 16:15:44.807534 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:15:44.807516 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277\": container with ID starting with 74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277 not found: ID does not exist" containerID="74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277" Jul 24 16:15:44.807572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.807545 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277"} err="failed to get container status \"74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277\": rpc error: code = NotFound desc = could not find container \"74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277\": container with ID starting with 74d617ebc07c41d748a395d6f013d34f8080697536c3a7ecaba8974864f71277 not found: ID does not exist" Jul 24 16:15:44.821000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.820976 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cpn6g"] Jul 24 16:15:44.825856 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:44.825833 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cpn6g"] Jul 24 16:15:45.166394 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:45.166363 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908feb44-5edc-4596-a8e2-5a771fc908aa" path="/var/lib/kubelet/pods/908feb44-5edc-4596-a8e2-5a771fc908aa/volumes" Jul 24 16:15:49.544445 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.544394 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:49.670299 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.670244 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:49.968968 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.968933 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf"] Jul 24 16:15:49.969527 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.969511 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908feb44-5edc-4596-a8e2-5a771fc908aa" containerName="limitador" Jul 24 16:15:49.969578 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.969532 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="908feb44-5edc-4596-a8e2-5a771fc908aa" containerName="limitador" Jul 24 16:15:49.969652 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.969641 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="908feb44-5edc-4596-a8e2-5a771fc908aa" containerName="limitador" Jul 24 16:15:49.973681 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.973662 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:15:49.976988 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.976966 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ddd7s\"" Jul 24 16:15:49.977112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.976966 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 16:15:49.977112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.977070 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 16:15:49.977699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.977681 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 16:15:49.992940 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:49.992915 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf"] Jul 24 16:15:50.028662 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.028638 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:15:50.092492 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.092465 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmz7\" (UniqueName: \"kubernetes.io/projected/31911853-3917-4c5f-a1e1-8b911098de4c-kube-api-access-4dmz7\") pod \"postgres-69fb4594cf-hn6cf\" (UID: \"31911853-3917-4c5f-a1e1-8b911098de4c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:15:50.092609 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.092504 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31911853-3917-4c5f-a1e1-8b911098de4c-data\") pod \"postgres-69fb4594cf-hn6cf\" (UID: \"31911853-3917-4c5f-a1e1-8b911098de4c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:15:50.193060 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.193039 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31911853-3917-4c5f-a1e1-8b911098de4c-data\") pod \"postgres-69fb4594cf-hn6cf\" (UID: \"31911853-3917-4c5f-a1e1-8b911098de4c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:15:50.193215 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.193104 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmz7\" (UniqueName: \"kubernetes.io/projected/31911853-3917-4c5f-a1e1-8b911098de4c-kube-api-access-4dmz7\") pod \"postgres-69fb4594cf-hn6cf\" (UID: \"31911853-3917-4c5f-a1e1-8b911098de4c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:15:50.193407 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.193390 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31911853-3917-4c5f-a1e1-8b911098de4c-data\") pod \"postgres-69fb4594cf-hn6cf\" (UID: \"31911853-3917-4c5f-a1e1-8b911098de4c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:15:50.201987 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.201959 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmz7\" (UniqueName: \"kubernetes.io/projected/31911853-3917-4c5f-a1e1-8b911098de4c-kube-api-access-4dmz7\") pod \"postgres-69fb4594cf-hn6cf\" (UID: \"31911853-3917-4c5f-a1e1-8b911098de4c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:15:50.284043 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.283960 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:15:50.415503 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.415452 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf"] Jul 24 16:15:50.417616 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:15:50.417588 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31911853_3917_4c5f_a1e1_8b911098de4c.slice/crio-5d0f8c12b4bbfb9be7fa47f7150aba23dfbcd2df430413bd61995be79329fe77 WatchSource:0}: Error finding container 5d0f8c12b4bbfb9be7fa47f7150aba23dfbcd2df430413bd61995be79329fe77: Status 404 returned error can't find the container with id 5d0f8c12b4bbfb9be7fa47f7150aba23dfbcd2df430413bd61995be79329fe77 Jul 24 16:15:50.830037 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:50.830004 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" event={"ID":"31911853-3917-4c5f-a1e1-8b911098de4c","Type":"ContainerStarted","Data":"5d0f8c12b4bbfb9be7fa47f7150aba23dfbcd2df430413bd61995be79329fe77"} Jul 24 16:15:55.858913 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:55.858875 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" event={"ID":"31911853-3917-4c5f-a1e1-8b911098de4c","Type":"ContainerStarted","Data":"1b331c4e283124b3dabe19ff5a531a99151c7526764d6d35743f07a13c9d8903"} Jul 24 16:15:55.859398 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:55.859007 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:15:55.878383 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:15:55.878334 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" podStartSLOduration=2.133800516 podStartE2EDuration="6.878318659s" totalImagesPullingTime="4.744518143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:15:55.877697826 +0000 UTC m=+575.309333188" watchObservedRunningTime="2026-07-24 16:15:55.878318659 +0000 UTC m=+575.309954020" Jul 24 16:16:01.856636 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:01.856605 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:01.891359 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:01.891334 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hn6cf" Jul 24 16:16:01.916478 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:01.916446 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:02.365802 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:02.365772 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:02.713221 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:02.713122 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:02.804235 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:02.804204 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:04.443357 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.443323 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wp95z"] Jul 24 16:16:04.443707 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.443505 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-wp95z" podUID="b62da17d-1c52-40ad-b3b5-03439e120ab2" containerName="authorino" containerID="cri-o://0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342" gracePeriod=30 Jul 24 16:16:04.758979 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.758954 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wp95z" Jul 24 16:16:04.901875 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.901834 2560 generic.go:356] "Generic (PLEG): container finished" podID="b62da17d-1c52-40ad-b3b5-03439e120ab2" containerID="0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342" exitCode=0 Jul 24 16:16:04.901875 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.901878 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wp95z" Jul 24 16:16:04.902078 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.901901 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wp95z" event={"ID":"b62da17d-1c52-40ad-b3b5-03439e120ab2","Type":"ContainerDied","Data":"0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342"} Jul 24 16:16:04.902078 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.901944 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wp95z" event={"ID":"b62da17d-1c52-40ad-b3b5-03439e120ab2","Type":"ContainerDied","Data":"b0ae637287c33f0f8e9224738053ad89f5bce3ad3518bc01de26b82ea3c2f254"} Jul 24 16:16:04.902078 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.901962 2560 scope.go:117] "RemoveContainer" containerID="0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342" Jul 24 16:16:04.910257 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.910241 2560 scope.go:117] "RemoveContainer" containerID="0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342" Jul 24 16:16:04.910505 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:16:04.910487 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342\": container with ID starting with 0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342 not found: ID does not exist" containerID="0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342" Jul 24 16:16:04.910565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.910518 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342"} err="failed to get container status \"0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342\": rpc error: code = NotFound desc = could not find container \"0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342\": container with ID starting with 0fbb21e044778a8daa27cca1f39f55a60eb38ba8fb93fa1488f6d11397c22342 not found: ID does not exist" Jul 24 16:16:04.922837 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.922812 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z9kp\" (UniqueName: \"kubernetes.io/projected/b62da17d-1c52-40ad-b3b5-03439e120ab2-kube-api-access-6z9kp\") pod \"b62da17d-1c52-40ad-b3b5-03439e120ab2\" (UID: \"b62da17d-1c52-40ad-b3b5-03439e120ab2\") " Jul 24 16:16:04.924955 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:04.924932 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62da17d-1c52-40ad-b3b5-03439e120ab2-kube-api-access-6z9kp" (OuterVolumeSpecName: "kube-api-access-6z9kp") pod "b62da17d-1c52-40ad-b3b5-03439e120ab2" (UID: "b62da17d-1c52-40ad-b3b5-03439e120ab2"). InnerVolumeSpecName "kube-api-access-6z9kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:16:05.023546 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:05.023485 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z9kp\" (UniqueName: \"kubernetes.io/projected/b62da17d-1c52-40ad-b3b5-03439e120ab2-kube-api-access-6z9kp\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:16:05.218694 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:05.218662 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wp95z"] Jul 24 16:16:05.233919 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:05.233897 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wp95z"] Jul 24 16:16:07.166000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:07.165958 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b62da17d-1c52-40ad-b3b5-03439e120ab2" path="/var/lib/kubelet/pods/b62da17d-1c52-40ad-b3b5-03439e120ab2/volumes" Jul 24 16:16:09.244241 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.244211 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:09.257485 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.257463 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7457ffcbb6-g7pn7"] Jul 24 16:16:09.257876 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.257862 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b62da17d-1c52-40ad-b3b5-03439e120ab2" containerName="authorino" Jul 24 16:16:09.257930 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.257878 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b62da17d-1c52-40ad-b3b5-03439e120ab2" containerName="authorino" Jul 24 16:16:09.257981 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.257972 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b62da17d-1c52-40ad-b3b5-03439e120ab2" containerName="authorino" Jul 24 16:16:09.272505 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.272477 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:09.273943 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.273908 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7457ffcbb6-g7pn7"] Jul 24 16:16:09.275366 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.275348 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7rbg5\"" Jul 24 16:16:09.275595 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.275554 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 16:16:09.275832 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.275813 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 16:16:09.330354 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.330326 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:09.431217 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.431186 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:09.442676 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.442648 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d8c774b47-bx2xr"] Jul 24 16:16:09.445050 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.445034 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:09.455632 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.455609 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-bx2xr"] Jul 24 16:16:09.460083 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.460061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwmz8\" (UniqueName: \"kubernetes.io/projected/649e1494-7f4a-4fba-b166-f999320d6a5d-kube-api-access-vwmz8\") pod \"maas-controller-7457ffcbb6-g7pn7\" (UID: \"649e1494-7f4a-4fba-b166-f999320d6a5d\") " pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:09.460203 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.460126 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/649e1494-7f4a-4fba-b166-f999320d6a5d-webhook-cert\") pod \"maas-controller-7457ffcbb6-g7pn7\" (UID: \"649e1494-7f4a-4fba-b166-f999320d6a5d\") " pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:09.561523 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.561447 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws97n\" (UniqueName: \"kubernetes.io/projected/319b14ef-f877-43f7-979d-b1d9bcc64c0f-kube-api-access-ws97n\") pod \"maas-controller-6d8c774b47-bx2xr\" (UID: \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\") " pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:09.561523 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.561482 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmz8\" (UniqueName: \"kubernetes.io/projected/649e1494-7f4a-4fba-b166-f999320d6a5d-kube-api-access-vwmz8\") pod \"maas-controller-7457ffcbb6-g7pn7\" (UID: \"649e1494-7f4a-4fba-b166-f999320d6a5d\") " pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:09.561697 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.561564 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/649e1494-7f4a-4fba-b166-f999320d6a5d-webhook-cert\") pod \"maas-controller-7457ffcbb6-g7pn7\" (UID: \"649e1494-7f4a-4fba-b166-f999320d6a5d\") " pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:09.561697 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.561622 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/319b14ef-f877-43f7-979d-b1d9bcc64c0f-webhook-cert\") pod \"maas-controller-6d8c774b47-bx2xr\" (UID: \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\") " pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:09.564194 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.564166 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/649e1494-7f4a-4fba-b166-f999320d6a5d-webhook-cert\") pod \"maas-controller-7457ffcbb6-g7pn7\" (UID: \"649e1494-7f4a-4fba-b166-f999320d6a5d\") " pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:09.572407 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.572387 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmz8\" (UniqueName: \"kubernetes.io/projected/649e1494-7f4a-4fba-b166-f999320d6a5d-kube-api-access-vwmz8\") pod \"maas-controller-7457ffcbb6-g7pn7\" (UID: \"649e1494-7f4a-4fba-b166-f999320d6a5d\") " pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:09.583267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.583246 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:09.662419 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.662382 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws97n\" (UniqueName: \"kubernetes.io/projected/319b14ef-f877-43f7-979d-b1d9bcc64c0f-kube-api-access-ws97n\") pod \"maas-controller-6d8c774b47-bx2xr\" (UID: \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\") " pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:09.662559 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.662494 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/319b14ef-f877-43f7-979d-b1d9bcc64c0f-webhook-cert\") pod \"maas-controller-6d8c774b47-bx2xr\" (UID: \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\") " pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:09.665411 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.665268 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/319b14ef-f877-43f7-979d-b1d9bcc64c0f-webhook-cert\") pod \"maas-controller-6d8c774b47-bx2xr\" (UID: \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\") " pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:09.672530 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.672505 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws97n\" (UniqueName: \"kubernetes.io/projected/319b14ef-f877-43f7-979d-b1d9bcc64c0f-kube-api-access-ws97n\") pod \"maas-controller-6d8c774b47-bx2xr\" (UID: \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\") " pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:09.714615 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.714590 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7457ffcbb6-g7pn7"] Jul 24 16:16:09.716302 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:16:09.716272 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649e1494_7f4a_4fba_b166_f999320d6a5d.slice/crio-3fd20ebd7e32739e237711c48d12af076b6f98e93f3f1e6b836cc1767a48328f WatchSource:0}: Error finding container 3fd20ebd7e32739e237711c48d12af076b6f98e93f3f1e6b836cc1767a48328f: Status 404 returned error can't find the container with id 3fd20ebd7e32739e237711c48d12af076b6f98e93f3f1e6b836cc1767a48328f Jul 24 16:16:09.757029 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.757001 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:09.888313 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.888222 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-bx2xr"] Jul 24 16:16:09.890538 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:16:09.890512 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319b14ef_f877_43f7_979d_b1d9bcc64c0f.slice/crio-3936fad8b8d92648308dbd27ed37e6be3e221dd70b7ac273eae0c26b6ce04f2a WatchSource:0}: Error finding container 3936fad8b8d92648308dbd27ed37e6be3e221dd70b7ac273eae0c26b6ce04f2a: Status 404 returned error can't find the container with id 3936fad8b8d92648308dbd27ed37e6be3e221dd70b7ac273eae0c26b6ce04f2a Jul 24 16:16:09.932622 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.932580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" event={"ID":"319b14ef-f877-43f7-979d-b1d9bcc64c0f","Type":"ContainerStarted","Data":"3936fad8b8d92648308dbd27ed37e6be3e221dd70b7ac273eae0c26b6ce04f2a"} Jul 24 16:16:09.933657 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:09.933635 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" event={"ID":"649e1494-7f4a-4fba-b166-f999320d6a5d","Type":"ContainerStarted","Data":"3fd20ebd7e32739e237711c48d12af076b6f98e93f3f1e6b836cc1767a48328f"} Jul 24 16:16:12.358781 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:12.358747 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:12.464988 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:12.464904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:12.953893 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:12.953856 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" event={"ID":"319b14ef-f877-43f7-979d-b1d9bcc64c0f","Type":"ContainerStarted","Data":"d6f2e20df8ef75443b9c4f0e5716f1858f10ccf88265cca4f852c95215f46117"} Jul 24 16:16:12.954092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:12.953926 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:12.955549 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:12.955524 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" event={"ID":"649e1494-7f4a-4fba-b166-f999320d6a5d","Type":"ContainerStarted","Data":"e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9"} Jul 24 16:16:12.955662 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:12.955604 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:13.000400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:13.000356 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" podStartSLOduration=1.7327172 podStartE2EDuration="4.000342663s" totalImagesPullingTime="2.267625463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:16:12.99903869 +0000 UTC m=+592.430674052" watchObservedRunningTime="2026-07-24 16:16:13.000342663 +0000 UTC m=+592.431978045" Jul 24 16:16:13.031631 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:13.031583 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" podStartSLOduration=1.584244714 podStartE2EDuration="4.031568419s" totalImagesPullingTime="2.447323705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:16:13.030022092 +0000 UTC m=+592.461657455" watchObservedRunningTime="2026-07-24 16:16:13.031568419 +0000 UTC m=+592.463203779" Jul 24 16:16:20.965195 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:20.965159 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:16:20.965999 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:20.965966 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:16:23.963156 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:23.963112 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:23.963523 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:23.963329 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:16:24.035805 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:24.035762 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7457ffcbb6-g7pn7"] Jul 24 16:16:24.035993 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:24.035970 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" podUID="649e1494-7f4a-4fba-b166-f999320d6a5d" containerName="manager" containerID="cri-o://e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9" gracePeriod=30 Jul 24 16:16:24.288704 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:24.288677 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:24.394046 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:24.394010 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwmz8\" (UniqueName: \"kubernetes.io/projected/649e1494-7f4a-4fba-b166-f999320d6a5d-kube-api-access-vwmz8\") pod \"649e1494-7f4a-4fba-b166-f999320d6a5d\" (UID: \"649e1494-7f4a-4fba-b166-f999320d6a5d\") " Jul 24 16:16:24.394263 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:24.394055 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/649e1494-7f4a-4fba-b166-f999320d6a5d-webhook-cert\") pod \"649e1494-7f4a-4fba-b166-f999320d6a5d\" (UID: \"649e1494-7f4a-4fba-b166-f999320d6a5d\") " Jul 24 16:16:24.396438 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:24.396413 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649e1494-7f4a-4fba-b166-f999320d6a5d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "649e1494-7f4a-4fba-b166-f999320d6a5d" (UID: "649e1494-7f4a-4fba-b166-f999320d6a5d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:16:24.396545 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:24.396436 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649e1494-7f4a-4fba-b166-f999320d6a5d-kube-api-access-vwmz8" (OuterVolumeSpecName: "kube-api-access-vwmz8") pod "649e1494-7f4a-4fba-b166-f999320d6a5d" (UID: "649e1494-7f4a-4fba-b166-f999320d6a5d"). InnerVolumeSpecName "kube-api-access-vwmz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:16:24.494843 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:24.494804 2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/649e1494-7f4a-4fba-b166-f999320d6a5d-webhook-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:16:24.494843 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:24.494838 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwmz8\" (UniqueName: \"kubernetes.io/projected/649e1494-7f4a-4fba-b166-f999320d6a5d-kube-api-access-vwmz8\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:16:25.017484 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.017454 2560 generic.go:356] "Generic (PLEG): container finished" podID="649e1494-7f4a-4fba-b166-f999320d6a5d" containerID="e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9" exitCode=0 Jul 24 16:16:25.017856 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.017508 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" Jul 24 16:16:25.017856 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.017543 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" event={"ID":"649e1494-7f4a-4fba-b166-f999320d6a5d","Type":"ContainerDied","Data":"e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9"} Jul 24 16:16:25.017856 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.017599 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457ffcbb6-g7pn7" event={"ID":"649e1494-7f4a-4fba-b166-f999320d6a5d","Type":"ContainerDied","Data":"3fd20ebd7e32739e237711c48d12af076b6f98e93f3f1e6b836cc1767a48328f"} Jul 24 16:16:25.017856 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.017627 2560 scope.go:117] "RemoveContainer" containerID="e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9" Jul 24 16:16:25.026913 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.026893 2560 scope.go:117] "RemoveContainer" containerID="e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9" Jul 24 16:16:25.027271 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:16:25.027248 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9\": container with ID starting with e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9 not found: ID does not exist" containerID="e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9" Jul 24 16:16:25.027330 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.027284 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9"} err="failed to get container status \"e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9\": rpc error: code = NotFound desc = could not find container \"e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9\": container with ID starting with e037580cd09c1180e7cdd1cf32ba3dc93cff44fc4d3ed4437f02335645777ae9 not found: ID does not exist" Jul 24 16:16:25.040127 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.040108 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7457ffcbb6-g7pn7"] Jul 24 16:16:25.045058 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.045035 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7457ffcbb6-g7pn7"] Jul 24 16:16:25.166162 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:25.166116 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="649e1494-7f4a-4fba-b166-f999320d6a5d" path="/var/lib/kubelet/pods/649e1494-7f4a-4fba-b166-f999320d6a5d/volumes" Jul 24 16:16:44.059572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.059491 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:44.716053 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.716020 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl"] Jul 24 16:16:44.716408 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.716395 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649e1494-7f4a-4fba-b166-f999320d6a5d" containerName="manager" Jul 24 16:16:44.716465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.716409 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="649e1494-7f4a-4fba-b166-f999320d6a5d" containerName="manager" Jul 24 16:16:44.716501 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.716483 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="649e1494-7f4a-4fba-b166-f999320d6a5d" containerName="manager" Jul 24 16:16:44.723974 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.723956 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:44.727031 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.727013 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 16:16:44.727227 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.727210 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w22br\"" Jul 24 16:16:44.733094 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.733060 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl"] Jul 24 16:16:44.872219 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.872191 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8mkl\" (UniqueName: \"kubernetes.io/projected/b4ed83a5-f630-48eb-bea4-fd4c5f5763c9-kube-api-access-c8mkl\") pod \"maas-api-d79565ccd-qpcvl\" (UID: \"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9\") " pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:44.872363 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.872227 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ed83a5-f630-48eb-bea4-fd4c5f5763c9-maas-api-tls\") pod \"maas-api-d79565ccd-qpcvl\" (UID: \"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9\") " pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:44.872363 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.872250 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ed83a5-f630-48eb-bea4-fd4c5f5763c9-openshift-service-ca\") pod \"maas-api-d79565ccd-qpcvl\" (UID: \"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9\") " pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:44.973611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.973531 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8mkl\" (UniqueName: \"kubernetes.io/projected/b4ed83a5-f630-48eb-bea4-fd4c5f5763c9-kube-api-access-c8mkl\") pod \"maas-api-d79565ccd-qpcvl\" (UID: \"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9\") " pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:44.973611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.973571 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ed83a5-f630-48eb-bea4-fd4c5f5763c9-maas-api-tls\") pod \"maas-api-d79565ccd-qpcvl\" (UID: \"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9\") " pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:44.973611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.973592 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ed83a5-f630-48eb-bea4-fd4c5f5763c9-openshift-service-ca\") pod \"maas-api-d79565ccd-qpcvl\" (UID: \"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9\") " pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:44.974259 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.974213 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ed83a5-f630-48eb-bea4-fd4c5f5763c9-openshift-service-ca\") pod \"maas-api-d79565ccd-qpcvl\" (UID: \"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9\") " pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:44.976005 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.975983 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ed83a5-f630-48eb-bea4-fd4c5f5763c9-maas-api-tls\") pod \"maas-api-d79565ccd-qpcvl\" (UID: \"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9\") " pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:44.984205 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:44.984183 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8mkl\" (UniqueName: \"kubernetes.io/projected/b4ed83a5-f630-48eb-bea4-fd4c5f5763c9-kube-api-access-c8mkl\") pod \"maas-api-d79565ccd-qpcvl\" (UID: \"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9\") " pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:45.037080 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.037046 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:45.182879 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.182855 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl"] Jul 24 16:16:45.184563 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:16:45.184538 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ed83a5_f630_48eb_bea4_fd4c5f5763c9.slice/crio-44bb6a44a7d45953d0a838688b9e47bb38121663e1dd6e3f571dce70db140bda WatchSource:0}: Error finding container 44bb6a44a7d45953d0a838688b9e47bb38121663e1dd6e3f571dce70db140bda: Status 404 returned error can't find the container with id 44bb6a44a7d45953d0a838688b9e47bb38121663e1dd6e3f571dce70db140bda Jul 24 16:16:45.226385 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.226282 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9l5g2"] Jul 24 16:16:45.234696 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.234666 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9l5g2"] Jul 24 16:16:45.234864 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.234773 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:16:45.237295 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.237275 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 16:16:45.237434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.237303 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-m7flr\"" Jul 24 16:16:45.377545 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.377515 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9cb2fb-a288-44d2-b490-f90c66e90120-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9l5g2\" (UID: \"6a9cb2fb-a288-44d2-b490-f90c66e90120\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:16:45.377545 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.377549 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59r26\" (UniqueName: \"kubernetes.io/projected/6a9cb2fb-a288-44d2-b490-f90c66e90120-kube-api-access-59r26\") pod \"payload-processing-5cc6cd444b-9l5g2\" (UID: \"6a9cb2fb-a288-44d2-b490-f90c66e90120\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:16:45.478664 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.478583 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9cb2fb-a288-44d2-b490-f90c66e90120-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9l5g2\" (UID: \"6a9cb2fb-a288-44d2-b490-f90c66e90120\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:16:45.478664 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.478625 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59r26\" (UniqueName: \"kubernetes.io/projected/6a9cb2fb-a288-44d2-b490-f90c66e90120-kube-api-access-59r26\") pod \"payload-processing-5cc6cd444b-9l5g2\" (UID: \"6a9cb2fb-a288-44d2-b490-f90c66e90120\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:16:45.479338 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.479320 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9cb2fb-a288-44d2-b490-f90c66e90120-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9l5g2\" (UID: \"6a9cb2fb-a288-44d2-b490-f90c66e90120\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:16:45.486849 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.486820 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59r26\" (UniqueName: \"kubernetes.io/projected/6a9cb2fb-a288-44d2-b490-f90c66e90120-kube-api-access-59r26\") pod \"payload-processing-5cc6cd444b-9l5g2\" (UID: \"6a9cb2fb-a288-44d2-b490-f90c66e90120\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:16:45.545816 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.545795 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:16:45.625810 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.625784 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:45.681875 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:45.681842 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9l5g2"] Jul 24 16:16:45.683298 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:16:45.683276 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9cb2fb_a288_44d2_b490_f90c66e90120.slice/crio-e847635e6362648615bb266490053a45e8b2e008aa41d8236aa13c72b073353a WatchSource:0}: Error finding container e847635e6362648615bb266490053a45e8b2e008aa41d8236aa13c72b073353a: Status 404 returned error can't find the container with id e847635e6362648615bb266490053a45e8b2e008aa41d8236aa13c72b073353a Jul 24 16:16:46.122757 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:46.122719 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" event={"ID":"6a9cb2fb-a288-44d2-b490-f90c66e90120","Type":"ContainerStarted","Data":"e847635e6362648615bb266490053a45e8b2e008aa41d8236aa13c72b073353a"} Jul 24 16:16:46.124238 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:46.124209 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" event={"ID":"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9","Type":"ContainerStarted","Data":"44bb6a44a7d45953d0a838688b9e47bb38121663e1dd6e3f571dce70db140bda"} Jul 24 16:16:48.136560 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:48.136528 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" event={"ID":"b4ed83a5-f630-48eb-bea4-fd4c5f5763c9","Type":"ContainerStarted","Data":"6704d38d208a6c460b68399d4e13fd73d6c99c99eea0ae0e3c3bb1a3787cc80a"} Jul 24 16:16:48.136995 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:48.136613 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:48.138100 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:48.138075 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" event={"ID":"6a9cb2fb-a288-44d2-b490-f90c66e90120","Type":"ContainerStarted","Data":"d2055bd175ee32152d9f90f67d63bb7e65264aacf256e12aa8c79aee82715f5f"} Jul 24 16:16:48.138207 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:48.138121 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:16:48.154043 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:48.153998 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" podStartSLOduration=1.942060021 podStartE2EDuration="4.153985434s" totalImagesPullingTime="2.211925413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:16:48.152105542 +0000 UTC m=+627.583740903" watchObservedRunningTime="2026-07-24 16:16:48.153985434 +0000 UTC m=+627.585620794" Jul 24 16:16:48.168928 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:48.168888 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" podStartSLOduration=1.448850464 podStartE2EDuration="3.168875086s" totalImagesPullingTime="1.720024622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:16:48.167358551 +0000 UTC m=+627.598993923" watchObservedRunningTime="2026-07-24 16:16:48.168875086 +0000 UTC m=+627.600510446" Jul 24 16:16:54.149841 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:54.149803 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d79565ccd-qpcvl" Jul 24 16:16:55.168598 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:55.168559 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:55.265785 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:55.265738 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:16:59.145983 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:16:59.145953 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9l5g2" Jul 24 16:17:00.050628 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.050596 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl"] Jul 24 16:17:00.054990 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.054970 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.057529 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.057509 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 16:17:00.058435 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.058410 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7bv6s\"" Jul 24 16:17:00.058579 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.058457 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 24 16:17:00.058579 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.058465 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 16:17:00.061579 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.061556 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl"] Jul 24 16:17:00.090255 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.090230 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:17:00.114562 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.114529 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.114710 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.114570 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cea08c2-8201-459d-b5be-70636e547578-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.114710 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.114682 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.114832 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.114713 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.114832 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.114754 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzw6\" (UniqueName: \"kubernetes.io/projected/4cea08c2-8201-459d-b5be-70636e547578-kube-api-access-gtzw6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.114832 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.114783 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.215434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.215406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.215434 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.215445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cea08c2-8201-459d-b5be-70636e547578-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.215953 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.215527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.215953 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.215564 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.215953 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.215626 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzw6\" (UniqueName: \"kubernetes.io/projected/4cea08c2-8201-459d-b5be-70636e547578-kube-api-access-gtzw6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.215953 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.215659 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.215953 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.215798 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.215953 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.215897 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.216282 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.216231 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.217983 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.217960 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cea08c2-8201-459d-b5be-70636e547578-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.218101 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.218060 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cea08c2-8201-459d-b5be-70636e547578-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.223112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.223090 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzw6\" (UniqueName: \"kubernetes.io/projected/4cea08c2-8201-459d-b5be-70636e547578-kube-api-access-gtzw6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl\" (UID: \"4cea08c2-8201-459d-b5be-70636e547578\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.368427 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.368403 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:00.503200 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:00.503175 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl"] Jul 24 16:17:00.505004 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:17:00.504977 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cea08c2_8201_459d_b5be_70636e547578.slice/crio-bbae4e4ad6a5d1729ace0d5143d5f1579dbc04cd98f0be2dded47ae8bdfe5413 WatchSource:0}: Error finding container bbae4e4ad6a5d1729ace0d5143d5f1579dbc04cd98f0be2dded47ae8bdfe5413: Status 404 returned error can't find the container with id bbae4e4ad6a5d1729ace0d5143d5f1579dbc04cd98f0be2dded47ae8bdfe5413 Jul 24 16:17:01.203306 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:01.203267 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" event={"ID":"4cea08c2-8201-459d-b5be-70636e547578","Type":"ContainerStarted","Data":"bbae4e4ad6a5d1729ace0d5143d5f1579dbc04cd98f0be2dded47ae8bdfe5413"} Jul 24 16:17:02.209234 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:02.209194 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" event={"ID":"4cea08c2-8201-459d-b5be-70636e547578","Type":"ContainerStarted","Data":"06b0ff969d9b4f83c00d8c88b82ef2eb92717743aeb8b5d076f8753c781f9d27"} Jul 24 16:17:02.229390 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:02.229345 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" podStartSLOduration=0.933709818 podStartE2EDuration="2.229330436s" totalImagesPullingTime="1.295620618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:17:02.227594663 +0000 UTC m=+641.659230025" watchObservedRunningTime="2026-07-24 16:17:02.229330436 +0000 UTC m=+641.660965798" Jul 24 16:17:03.432312 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:03.432286 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:17:10.369485 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:10.369451 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:10.369824 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:10.369521 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:10.382012 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:10.381989 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:11.232632 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:11.232597 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:17:11.262120 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:11.262091 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl" Jul 24 16:17:22.783169 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:22.783114 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:17:33.249923 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.249892 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm"] Jul 24 16:17:33.256498 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.256478 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.259254 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.259233 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 16:17:33.263611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.263590 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm"] Jul 24 16:17:33.342608 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.342578 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:17:33.426991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.426961 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkr6h\" (UniqueName: \"kubernetes.io/projected/6d8d03ae-516b-4d87-8429-e1f111d41915-kube-api-access-mkr6h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.427131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.427024 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.427131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.427046 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8d03ae-516b-4d87-8429-e1f111d41915-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.427131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.427085 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.427256 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.427173 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.427256 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.427228 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.528572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.528496 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkr6h\" (UniqueName: \"kubernetes.io/projected/6d8d03ae-516b-4d87-8429-e1f111d41915-kube-api-access-mkr6h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.528572 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.528552 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.528743 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.528715 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8d03ae-516b-4d87-8429-e1f111d41915-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.528805 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.528788 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.529000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.528826 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.529000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.528849 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.529000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.528899 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.529289 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.529231 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.529289 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.529250 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.530983 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.530965 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8d03ae-516b-4d87-8429-e1f111d41915-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.531248 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.531231 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8d03ae-516b-4d87-8429-e1f111d41915-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.541046 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.541026 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkr6h\" (UniqueName: \"kubernetes.io/projected/6d8d03ae-516b-4d87-8429-e1f111d41915-kube-api-access-mkr6h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm\" (UID: \"6d8d03ae-516b-4d87-8429-e1f111d41915\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.568044 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.568017 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:33.705912 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.705888 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm"] Jul 24 16:17:33.707567 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:17:33.707544 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8d03ae_516b_4d87_8429_e1f111d41915.slice/crio-5e99d3db33380be632bb49e4d6aed66c0fce943783bfd456448915cb671e5860 WatchSource:0}: Error finding container 5e99d3db33380be632bb49e4d6aed66c0fce943783bfd456448915cb671e5860: Status 404 returned error can't find the container with id 5e99d3db33380be632bb49e4d6aed66c0fce943783bfd456448915cb671e5860 Jul 24 16:17:33.708930 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:33.708910 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:17:34.362968 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:34.362930 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" event={"ID":"6d8d03ae-516b-4d87-8429-e1f111d41915","Type":"ContainerStarted","Data":"c48ed5b43e9e9f3f28811cd830c788c30abf8d12e7377405110a0671c1c1a971"} Jul 24 16:17:34.363435 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:34.362979 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" event={"ID":"6d8d03ae-516b-4d87-8429-e1f111d41915","Type":"ContainerStarted","Data":"5e99d3db33380be632bb49e4d6aed66c0fce943783bfd456448915cb671e5860"} Jul 24 16:17:34.380813 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:34.380771 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" podStartSLOduration=1.115045032 podStartE2EDuration="1.380755523s" totalImagesPullingTime="265.710491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:17:34.379871052 +0000 UTC m=+673.811506413" watchObservedRunningTime="2026-07-24 16:17:34.380755523 +0000 UTC m=+673.812390883" Jul 24 16:17:38.736550 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:38.736520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:17:38.814959 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:38.814931 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:17:43.568368 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:43.568333 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:43.568752 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:43.568409 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:43.580942 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:43.580918 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:44.422319 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:44.422289 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm" Jul 24 16:17:48.282955 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:48.282916 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:17:50.450257 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:17:50.450225 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:18:56.396825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.396794 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b78dd6555-4wjhj"] Jul 24 16:18:56.400765 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.400747 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b78dd6555-4wjhj" Jul 24 16:18:56.403971 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.403955 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w5pns\"" Jul 24 16:18:56.404061 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.404001 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 24 16:18:56.407832 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.407810 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b78dd6555-4wjhj"] Jul 24 16:18:56.467979 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.467954 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd62a736-38ba-4386-9af1-1497e68ae17d-tls-cert\") pod \"authorino-b78dd6555-4wjhj\" (UID: \"dd62a736-38ba-4386-9af1-1497e68ae17d\") " pod="kuadrant-system/authorino-b78dd6555-4wjhj" Jul 24 16:18:56.468131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.467995 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjb2\" (UniqueName: \"kubernetes.io/projected/dd62a736-38ba-4386-9af1-1497e68ae17d-kube-api-access-hqjb2\") pod \"authorino-b78dd6555-4wjhj\" (UID: \"dd62a736-38ba-4386-9af1-1497e68ae17d\") " pod="kuadrant-system/authorino-b78dd6555-4wjhj" Jul 24 16:18:56.568821 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.568794 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd62a736-38ba-4386-9af1-1497e68ae17d-tls-cert\") pod \"authorino-b78dd6555-4wjhj\" (UID: \"dd62a736-38ba-4386-9af1-1497e68ae17d\") " pod="kuadrant-system/authorino-b78dd6555-4wjhj" Jul 24 16:18:56.568965 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.568836 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjb2\" (UniqueName: \"kubernetes.io/projected/dd62a736-38ba-4386-9af1-1497e68ae17d-kube-api-access-hqjb2\") pod \"authorino-b78dd6555-4wjhj\" (UID: \"dd62a736-38ba-4386-9af1-1497e68ae17d\") " pod="kuadrant-system/authorino-b78dd6555-4wjhj" Jul 24 16:18:56.571456 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.571422 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd62a736-38ba-4386-9af1-1497e68ae17d-tls-cert\") pod \"authorino-b78dd6555-4wjhj\" (UID: \"dd62a736-38ba-4386-9af1-1497e68ae17d\") " pod="kuadrant-system/authorino-b78dd6555-4wjhj" Jul 24 16:18:56.576718 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.576696 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjb2\" (UniqueName: \"kubernetes.io/projected/dd62a736-38ba-4386-9af1-1497e68ae17d-kube-api-access-hqjb2\") pod \"authorino-b78dd6555-4wjhj\" (UID: \"dd62a736-38ba-4386-9af1-1497e68ae17d\") " pod="kuadrant-system/authorino-b78dd6555-4wjhj" Jul 24 16:18:56.711485 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.711417 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b78dd6555-4wjhj" Jul 24 16:18:56.722931 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.722906 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:18:56.816553 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.816513 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:18:56.849184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:56.849139 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b78dd6555-4wjhj"] Jul 24 16:18:56.851684 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:18:56.851660 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd62a736_38ba_4386_9af1_1497e68ae17d.slice/crio-d7f710002f71df9b3098309fa68ad2e0e60b98936933daea99f108bc431fbcf6 WatchSource:0}: Error finding container d7f710002f71df9b3098309fa68ad2e0e60b98936933daea99f108bc431fbcf6: Status 404 returned error can't find the container with id d7f710002f71df9b3098309fa68ad2e0e60b98936933daea99f108bc431fbcf6 Jul 24 16:18:57.779131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:57.779076 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b78dd6555-4wjhj" event={"ID":"dd62a736-38ba-4386-9af1-1497e68ae17d","Type":"ContainerStarted","Data":"a1419a49a27def14944622b31aa1530f414814fad858e5e62e59d49bb60eaf08"} Jul 24 16:18:57.779614 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:57.779561 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b78dd6555-4wjhj" event={"ID":"dd62a736-38ba-4386-9af1-1497e68ae17d","Type":"ContainerStarted","Data":"d7f710002f71df9b3098309fa68ad2e0e60b98936933daea99f108bc431fbcf6"} Jul 24 16:18:57.804370 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:57.804314 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b78dd6555-4wjhj" podStartSLOduration=1.3510792440000001 podStartE2EDuration="1.804298423s" totalImagesPullingTime="453.219179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:18:57.802239145 +0000 UTC m=+757.233874505" watchObservedRunningTime="2026-07-24 16:18:57.804298423 +0000 UTC m=+757.235933782" Jul 24 16:18:57.849196 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:57.849132 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:18:57.947589 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:18:57.947559 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:19:10.597909 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:10.597879 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:19:43.806926 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:43.806823 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-bx2xr"] Jul 24 16:19:43.807508 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:43.807084 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" podUID="319b14ef-f877-43f7-979d-b1d9bcc64c0f" containerName="manager" containerID="cri-o://d6f2e20df8ef75443b9c4f0e5716f1858f10ccf88265cca4f852c95215f46117" gracePeriod=30 Jul 24 16:19:43.961598 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:43.961548 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" podUID="319b14ef-f877-43f7-979d-b1d9bcc64c0f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.37:8081/readyz\": dial tcp 10.134.0.37:8081: connect: connection refused" Jul 24 16:19:44.018290 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:44.018260 2560 generic.go:356] "Generic (PLEG): container finished" podID="319b14ef-f877-43f7-979d-b1d9bcc64c0f" containerID="d6f2e20df8ef75443b9c4f0e5716f1858f10ccf88265cca4f852c95215f46117" exitCode=0 Jul 24 16:19:44.018443 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:44.018326 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" event={"ID":"319b14ef-f877-43f7-979d-b1d9bcc64c0f","Type":"ContainerDied","Data":"d6f2e20df8ef75443b9c4f0e5716f1858f10ccf88265cca4f852c95215f46117"} Jul 24 16:19:44.051640 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:44.051615 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:19:44.213765 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:44.213738 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws97n\" (UniqueName: \"kubernetes.io/projected/319b14ef-f877-43f7-979d-b1d9bcc64c0f-kube-api-access-ws97n\") pod \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\" (UID: \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\") " Jul 24 16:19:44.213930 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:44.213776 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/319b14ef-f877-43f7-979d-b1d9bcc64c0f-webhook-cert\") pod \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\" (UID: \"319b14ef-f877-43f7-979d-b1d9bcc64c0f\") " Jul 24 16:19:44.215856 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:44.215827 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319b14ef-f877-43f7-979d-b1d9bcc64c0f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "319b14ef-f877-43f7-979d-b1d9bcc64c0f" (UID: "319b14ef-f877-43f7-979d-b1d9bcc64c0f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:19:44.215970 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:44.215898 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319b14ef-f877-43f7-979d-b1d9bcc64c0f-kube-api-access-ws97n" (OuterVolumeSpecName: "kube-api-access-ws97n") pod "319b14ef-f877-43f7-979d-b1d9bcc64c0f" (UID: "319b14ef-f877-43f7-979d-b1d9bcc64c0f"). InnerVolumeSpecName "kube-api-access-ws97n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:19:44.314891 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:44.314866 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws97n\" (UniqueName: \"kubernetes.io/projected/319b14ef-f877-43f7-979d-b1d9bcc64c0f-kube-api-access-ws97n\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:19:44.315023 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:44.314895 2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/319b14ef-f877-43f7-979d-b1d9bcc64c0f-webhook-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:19:45.024804 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:45.024774 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" Jul 24 16:19:45.024804 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:45.024800 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-bx2xr" event={"ID":"319b14ef-f877-43f7-979d-b1d9bcc64c0f","Type":"ContainerDied","Data":"3936fad8b8d92648308dbd27ed37e6be3e221dd70b7ac273eae0c26b6ce04f2a"} Jul 24 16:19:45.025385 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:45.024856 2560 scope.go:117] "RemoveContainer" containerID="d6f2e20df8ef75443b9c4f0e5716f1858f10ccf88265cca4f852c95215f46117" Jul 24 16:19:45.045922 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:45.045898 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-bx2xr"] Jul 24 16:19:45.049305 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:45.049277 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-bx2xr"] Jul 24 16:19:45.170011 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:19:45.169978 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="319b14ef-f877-43f7-979d-b1d9bcc64c0f" path="/var/lib/kubelet/pods/319b14ef-f877-43f7-979d-b1d9bcc64c0f/volumes" Jul 24 16:20:45.864260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.864227 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d8c774b47-qxngk"] Jul 24 16:20:45.864660 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.864619 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319b14ef-f877-43f7-979d-b1d9bcc64c0f" containerName="manager" Jul 24 16:20:45.864660 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.864629 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="319b14ef-f877-43f7-979d-b1d9bcc64c0f" containerName="manager" Jul 24 16:20:45.864738 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.864701 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="319b14ef-f877-43f7-979d-b1d9bcc64c0f" containerName="manager" Jul 24 16:20:45.867688 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.867673 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:20:45.871087 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.871065 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 16:20:45.871226 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.871089 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 16:20:45.871226 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.871094 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7rbg5\"" Jul 24 16:20:45.876590 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.876568 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-qxngk"] Jul 24 16:20:45.995827 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.995796 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtfq\" (UniqueName: \"kubernetes.io/projected/69c4ad5b-b32f-4b42-bee6-d4669f07555d-kube-api-access-lbtfq\") pod \"maas-controller-6d8c774b47-qxngk\" (UID: \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\") " pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:20:45.996061 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:45.995858 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69c4ad5b-b32f-4b42-bee6-d4669f07555d-webhook-cert\") pod \"maas-controller-6d8c774b47-qxngk\" (UID: \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\") " pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:20:46.096665 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:46.096636 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtfq\" (UniqueName: \"kubernetes.io/projected/69c4ad5b-b32f-4b42-bee6-d4669f07555d-kube-api-access-lbtfq\") pod \"maas-controller-6d8c774b47-qxngk\" (UID: \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\") " pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:20:46.096823 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:46.096691 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69c4ad5b-b32f-4b42-bee6-d4669f07555d-webhook-cert\") pod \"maas-controller-6d8c774b47-qxngk\" (UID: \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\") " pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:20:46.099409 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:46.099387 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69c4ad5b-b32f-4b42-bee6-d4669f07555d-webhook-cert\") pod \"maas-controller-6d8c774b47-qxngk\" (UID: \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\") " pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:20:46.105054 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:46.105031 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtfq\" (UniqueName: \"kubernetes.io/projected/69c4ad5b-b32f-4b42-bee6-d4669f07555d-kube-api-access-lbtfq\") pod \"maas-controller-6d8c774b47-qxngk\" (UID: \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\") " pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:20:46.179068 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:46.179004 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:20:46.308981 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:46.308960 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-qxngk"] Jul 24 16:20:46.310829 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:20:46.310805 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c4ad5b_b32f_4b42_bee6_d4669f07555d.slice/crio-d6906b1bf6a2b064ba9576df61cd302b58fc69f2ffe30a755554f33db1d55c2a WatchSource:0}: Error finding container d6906b1bf6a2b064ba9576df61cd302b58fc69f2ffe30a755554f33db1d55c2a: Status 404 returned error can't find the container with id d6906b1bf6a2b064ba9576df61cd302b58fc69f2ffe30a755554f33db1d55c2a Jul 24 16:20:46.323660 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:46.323636 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-qxngk" event={"ID":"69c4ad5b-b32f-4b42-bee6-d4669f07555d","Type":"ContainerStarted","Data":"d6906b1bf6a2b064ba9576df61cd302b58fc69f2ffe30a755554f33db1d55c2a"} Jul 24 16:20:47.331203 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:47.331176 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-qxngk" event={"ID":"69c4ad5b-b32f-4b42-bee6-d4669f07555d","Type":"ContainerStarted","Data":"36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b"} Jul 24 16:20:47.331857 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:47.331290 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:20:47.351375 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:47.351331 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d8c774b47-qxngk" podStartSLOduration=1.857271093 podStartE2EDuration="2.351315877s" totalImagesPullingTime="494.044784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:20:47.350458475 +0000 UTC m=+866.782093828" watchObservedRunningTime="2026-07-24 16:20:47.351315877 +0000 UTC m=+866.782951237" Jul 24 16:20:58.339114 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:20:58.339086 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:21:02.506642 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:02.506588 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:02.875976 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:02.875947 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:02.977051 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:02.977023 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-qxngk"] Jul 24 16:21:02.977325 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:02.977293 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d8c774b47-qxngk" podUID="69c4ad5b-b32f-4b42-bee6-d4669f07555d" containerName="manager" containerID="cri-o://36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b" gracePeriod=30 Jul 24 16:21:03.222639 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.222617 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:21:03.359735 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.359710 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbtfq\" (UniqueName: \"kubernetes.io/projected/69c4ad5b-b32f-4b42-bee6-d4669f07555d-kube-api-access-lbtfq\") pod \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\" (UID: \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\") " Jul 24 16:21:03.359891 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.359756 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69c4ad5b-b32f-4b42-bee6-d4669f07555d-webhook-cert\") pod \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\" (UID: \"69c4ad5b-b32f-4b42-bee6-d4669f07555d\") " Jul 24 16:21:03.362021 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.361993 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c4ad5b-b32f-4b42-bee6-d4669f07555d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "69c4ad5b-b32f-4b42-bee6-d4669f07555d" (UID: "69c4ad5b-b32f-4b42-bee6-d4669f07555d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:21:03.362134 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.361996 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c4ad5b-b32f-4b42-bee6-d4669f07555d-kube-api-access-lbtfq" (OuterVolumeSpecName: "kube-api-access-lbtfq") pod "69c4ad5b-b32f-4b42-bee6-d4669f07555d" (UID: "69c4ad5b-b32f-4b42-bee6-d4669f07555d"). InnerVolumeSpecName "kube-api-access-lbtfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:21:03.411936 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.411868 2560 generic.go:356] "Generic (PLEG): container finished" podID="69c4ad5b-b32f-4b42-bee6-d4669f07555d" containerID="36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b" exitCode=0 Jul 24 16:21:03.411936 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.411911 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-qxngk" Jul 24 16:21:03.411936 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.411928 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-qxngk" event={"ID":"69c4ad5b-b32f-4b42-bee6-d4669f07555d","Type":"ContainerDied","Data":"36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b"} Jul 24 16:21:03.412115 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.411969 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-qxngk" event={"ID":"69c4ad5b-b32f-4b42-bee6-d4669f07555d","Type":"ContainerDied","Data":"d6906b1bf6a2b064ba9576df61cd302b58fc69f2ffe30a755554f33db1d55c2a"} Jul 24 16:21:03.412115 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.411989 2560 scope.go:117] "RemoveContainer" containerID="36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b" Jul 24 16:21:03.423101 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.421122 2560 scope.go:117] "RemoveContainer" containerID="36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b" Jul 24 16:21:03.423477 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:21:03.423454 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b\": container with ID starting with 36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b not found: ID does not exist" containerID="36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b" Jul 24 16:21:03.423565 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.423490 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b"} err="failed to get container status \"36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b\": rpc error: code = NotFound desc = could not find container \"36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b\": container with ID starting with 36bc23e01777c42b21c99938e2ebc60c4c0d616dc57ad5b43a3669a3a92de07b not found: ID does not exist" Jul 24 16:21:03.431971 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.431948 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-qxngk"] Jul 24 16:21:03.435979 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.435958 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-qxngk"] Jul 24 16:21:03.461002 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.460981 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtfq\" (UniqueName: \"kubernetes.io/projected/69c4ad5b-b32f-4b42-bee6-d4669f07555d-kube-api-access-lbtfq\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:21:03.461002 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:03.461002 2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69c4ad5b-b32f-4b42-bee6-d4669f07555d-webhook-cert\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:21:04.119296 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.119265 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:04.233550 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.233522 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d8c774b47-9mrg4"] Jul 24 16:21:04.233925 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.233914 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c4ad5b-b32f-4b42-bee6-d4669f07555d" containerName="manager" Jul 24 16:21:04.233969 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.233927 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c4ad5b-b32f-4b42-bee6-d4669f07555d" containerName="manager" Jul 24 16:21:04.234004 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.233983 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c4ad5b-b32f-4b42-bee6-d4669f07555d" containerName="manager" Jul 24 16:21:04.238295 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.238277 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:04.241564 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.241541 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 16:21:04.241682 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.241576 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 16:21:04.241682 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.241547 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7rbg5\"" Jul 24 16:21:04.245958 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.245937 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-9mrg4"] Jul 24 16:21:04.371000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.370933 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b561f2fa-cb00-4c0e-8e69-eac1419f35a1-webhook-cert\") pod \"maas-controller-6d8c774b47-9mrg4\" (UID: \"b561f2fa-cb00-4c0e-8e69-eac1419f35a1\") " pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:04.371123 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.371075 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9jw\" (UniqueName: \"kubernetes.io/projected/b561f2fa-cb00-4c0e-8e69-eac1419f35a1-kube-api-access-wc9jw\") pod \"maas-controller-6d8c774b47-9mrg4\" (UID: \"b561f2fa-cb00-4c0e-8e69-eac1419f35a1\") " pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:04.471996 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.471960 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9jw\" (UniqueName: \"kubernetes.io/projected/b561f2fa-cb00-4c0e-8e69-eac1419f35a1-kube-api-access-wc9jw\") pod \"maas-controller-6d8c774b47-9mrg4\" (UID: \"b561f2fa-cb00-4c0e-8e69-eac1419f35a1\") " pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:04.472184 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.472030 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b561f2fa-cb00-4c0e-8e69-eac1419f35a1-webhook-cert\") pod \"maas-controller-6d8c774b47-9mrg4\" (UID: \"b561f2fa-cb00-4c0e-8e69-eac1419f35a1\") " pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:04.474611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.474593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b561f2fa-cb00-4c0e-8e69-eac1419f35a1-webhook-cert\") pod \"maas-controller-6d8c774b47-9mrg4\" (UID: \"b561f2fa-cb00-4c0e-8e69-eac1419f35a1\") " pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:04.479743 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.479723 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9jw\" (UniqueName: \"kubernetes.io/projected/b561f2fa-cb00-4c0e-8e69-eac1419f35a1-kube-api-access-wc9jw\") pod \"maas-controller-6d8c774b47-9mrg4\" (UID: \"b561f2fa-cb00-4c0e-8e69-eac1419f35a1\") " pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:04.571030 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.571000 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:04.701936 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:04.701912 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8c774b47-9mrg4"] Jul 24 16:21:04.703964 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:21:04.703943 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb561f2fa_cb00_4c0e_8e69_eac1419f35a1.slice/crio-4054a5f381d95f6caa16c1bdab730ccea2b652f84fe58a66630bf8b59aa2a0b7 WatchSource:0}: Error finding container 4054a5f381d95f6caa16c1bdab730ccea2b652f84fe58a66630bf8b59aa2a0b7: Status 404 returned error can't find the container with id 4054a5f381d95f6caa16c1bdab730ccea2b652f84fe58a66630bf8b59aa2a0b7 Jul 24 16:21:05.167259 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:05.167231 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c4ad5b-b32f-4b42-bee6-d4669f07555d" path="/var/lib/kubelet/pods/69c4ad5b-b32f-4b42-bee6-d4669f07555d/volumes" Jul 24 16:21:05.424892 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:05.424862 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-9mrg4" event={"ID":"b561f2fa-cb00-4c0e-8e69-eac1419f35a1","Type":"ContainerStarted","Data":"af7b9d1e31941e53fdb0252adad367c3e16765b545dba0bed27570aff22f6194"} Jul 24 16:21:05.425061 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:05.424911 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8c774b47-9mrg4" event={"ID":"b561f2fa-cb00-4c0e-8e69-eac1419f35a1","Type":"ContainerStarted","Data":"4054a5f381d95f6caa16c1bdab730ccea2b652f84fe58a66630bf8b59aa2a0b7"} Jul 24 16:21:05.425061 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:05.424997 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:05.446712 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:05.446665 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d8c774b47-9mrg4" podStartSLOduration=1.086907244 podStartE2EDuration="1.446651786s" totalImagesPullingTime="359.744542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:21:05.442300305 +0000 UTC m=+884.873935675" watchObservedRunningTime="2026-07-24 16:21:05.446651786 +0000 UTC m=+884.878287146" Jul 24 16:21:16.431773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:16.431746 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d8c774b47-9mrg4" Jul 24 16:21:20.999243 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:20.999205 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:21:21.000588 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:21.000567 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:21:46.852581 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:46.852545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:48.407050 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:48.407018 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:49.771778 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:49.771743 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:52.834325 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:52.834288 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:52.962165 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:52.962116 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:53.060135 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:53.060100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:53.190379 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:53.190345 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:58.811310 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:58.811276 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:21:58.897969 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:21:58.897925 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:22:12.829751 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:22:12.829716 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:26:21.030436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:26:21.030405 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:26:21.032399 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:26:21.032379 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:26:54.049925 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:26:54.049891 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:26:55.290961 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:26:55.290931 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:27:23.312759 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:27:23.312726 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:29:25.281665 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:29:25.281625 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:29:42.969967 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:29:42.969932 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:30:27.885571 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:30:27.885530 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:30:29.396881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:30:29.396847 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:31:21.062179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:31:21.062129 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:31:21.065531 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:31:21.065510 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:32:11.943498 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:32:11.943437 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:32:21.901379 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:32:21.901344 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:33:33.666751 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:33:33.666704 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:33:59.342041 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:33:59.342011 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:02.355205 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:02.355166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:02.456782 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:02.456748 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:02.813990 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:02.813950 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:08.419386 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:08.419349 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:08.481955 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:08.481895 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:08.644960 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:08.644923 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:08.952669 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:08.952637 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:09.698857 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:09.698808 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:11.324073 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:11.324045 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:11.430158 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:11.430123 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:11.547276 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:11.547241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:11.965083 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:11.965051 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:12.066172 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:12.066099 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:12.149371 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:12.149337 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:47.331928 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:47.331844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:47.410777 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:47.410747 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:47.470254 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:47.470220 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:51.930008 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:51.929973 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:52.156612 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:52.156545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:52.348689 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:52.348576 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:52.619321 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:52.619265 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:52.705897 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:52.705867 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:52.851838 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:52.851805 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:53.135221 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:53.135177 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:53.356942 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:53.356909 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:55.702966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:55.702936 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:55.819347 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:55.819318 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:55.915487 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:55.915450 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:56.365545 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:56.365510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:34:56.461265 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:34:56.461204 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:31.152164 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:31.152105 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:31.219706 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:31.219673 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:31.286193 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:31.286156 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:35.775091 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:35.775049 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:35.935287 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:35.935242 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:36.182911 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:36.182875 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:36.408278 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:36.408246 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:36.741407 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:36.741371 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:37.395016 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:37.394986 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:38.847437 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:38.847385 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:38.951944 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:38.951914 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:39.123058 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:39.123027 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:39.258394 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:39.258363 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:39.650541 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:39.650513 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:39.739442 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:39.739404 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:35:45.100297 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:35:45.100250 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:04.667548 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:04.667512 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6"] Jul 24 16:36:04.670995 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:04.670978 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" Jul 24 16:36:04.673482 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:04.673456 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtjsm\"" Jul 24 16:36:04.677002 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:04.676981 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6"] Jul 24 16:36:04.833126 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:04.833088 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437f9fb-b260-4cbe-91de-050a39665adf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c399289-j48l6\" (UID: \"5437f9fb-b260-4cbe-91de-050a39665adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" Jul 24 16:36:04.934368 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:04.934280 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437f9fb-b260-4cbe-91de-050a39665adf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c399289-j48l6\" (UID: \"5437f9fb-b260-4cbe-91de-050a39665adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" Jul 24 16:36:04.934958 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:04.934939 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437f9fb-b260-4cbe-91de-050a39665adf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c399289-j48l6\" (UID: \"5437f9fb-b260-4cbe-91de-050a39665adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" Jul 24 16:36:04.983461 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:04.983424 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" Jul 24 16:36:05.142204 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:05.142096 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6"] Jul 24 16:36:05.145369 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:36:05.145340 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5437f9fb_b260_4cbe_91de_050a39665adf.slice/crio-b99fb3eaf87cf504ab34754e18bfeacee12126dd0e8f382786be94cd741b0565 WatchSource:0}: Error finding container b99fb3eaf87cf504ab34754e18bfeacee12126dd0e8f382786be94cd741b0565: Status 404 returned error can't find the container with id b99fb3eaf87cf504ab34754e18bfeacee12126dd0e8f382786be94cd741b0565 Jul 24 16:36:05.149543 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:05.149528 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:36:05.933887 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:05.933851 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" event={"ID":"5437f9fb-b260-4cbe-91de-050a39665adf","Type":"ContainerStarted","Data":"b99fb3eaf87cf504ab34754e18bfeacee12126dd0e8f382786be94cd741b0565"} Jul 24 16:36:08.951774 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:08.951690 2560 generic.go:356] "Generic (PLEG): container finished" podID="5437f9fb-b260-4cbe-91de-050a39665adf" containerID="76ddc6c302e8fe1b615e3d87c78c7a78552600a799ec1355089ca82531109e9a" exitCode=0 Jul 24 16:36:08.952118 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:08.951777 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" event={"ID":"5437f9fb-b260-4cbe-91de-050a39665adf","Type":"ContainerDied","Data":"76ddc6c302e8fe1b615e3d87c78c7a78552600a799ec1355089ca82531109e9a"} Jul 24 16:36:10.091179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:10.091134 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" Jul 24 16:36:10.183193 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:10.183140 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437f9fb-b260-4cbe-91de-050a39665adf-maas-api-service-ca\") pod \"5437f9fb-b260-4cbe-91de-050a39665adf\" (UID: \"5437f9fb-b260-4cbe-91de-050a39665adf\") " Jul 24 16:36:10.183445 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:10.183423 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5437f9fb-b260-4cbe-91de-050a39665adf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5437f9fb-b260-4cbe-91de-050a39665adf" (UID: "5437f9fb-b260-4cbe-91de-050a39665adf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:36:10.284645 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:10.284551 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437f9fb-b260-4cbe-91de-050a39665adf-maas-api-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:36:10.963280 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:10.963234 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" event={"ID":"5437f9fb-b260-4cbe-91de-050a39665adf","Type":"ContainerDied","Data":"b99fb3eaf87cf504ab34754e18bfeacee12126dd0e8f382786be94cd741b0565"} Jul 24 16:36:10.963453 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:10.963291 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b99fb3eaf87cf504ab34754e18bfeacee12126dd0e8f382786be94cd741b0565" Jul 24 16:36:10.963453 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:10.963254 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6" Jul 24 16:36:15.120680 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:15.120629 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:15.221363 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:15.221316 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:15.278205 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:15.278172 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:19.755246 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:19.755129 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6"] Jul 24 16:36:19.757353 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:19.757331 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:19.757978 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:19.757958 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c399289-j48l6"] Jul 24 16:36:20.942553 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:20.942491 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:21.094376 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:21.094342 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:36:21.098215 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:21.098192 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:36:21.160563 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:21.160526 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:21.166688 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:21.166662 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5437f9fb-b260-4cbe-91de-050a39665adf" path="/var/lib/kubelet/pods/5437f9fb-b260-4cbe-91de-050a39665adf/volumes" Jul 24 16:36:21.386946 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:21.386907 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:21.745681 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:21.745593 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:21.906305 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:21.906271 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:24.149888 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:24.149852 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:24.242824 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:24.242795 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:24.367513 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:24.367486 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:24.565533 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:24.565434 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:24.841582 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:24.841483 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:24.941265 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:24.941232 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:45.286330 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:45.286287 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:59.509622 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:59.509582 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:59.647808 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:59.647766 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:36:59.736879 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:36:59.736851 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:04.106182 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:04.106133 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:04.269609 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:04.269566 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:04.508576 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:04.508545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:04.972894 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:04.972864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:05.078579 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:05.078545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:05.177161 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:05.177117 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:06.590002 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:06.589966 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:12.531382 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:12.531352 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:12.639553 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:12.639513 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:12.731653 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:12.731605 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:12.840747 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:12.840657 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:13.234416 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:13.234379 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:39.448301 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:39.448260 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:45.182498 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:45.182417 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:45.283312 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:45.283273 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:37:45.383744 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:37:45.383698 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:01.214801 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:01.214770 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:07.269695 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:07.269643 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:07.329239 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:07.329178 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:07.418803 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:07.418769 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:07.521192 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:07.521058 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:17.996695 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:17.996652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:18.773600 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:18.773548 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:25.235965 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:25.235932 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:25.372666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:25.372635 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:25.477003 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:25.476968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:25.573071 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:25.572993 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:26.323600 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:26.323570 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:26.454388 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:26.454358 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:26.556464 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:26.556428 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:28.105374 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:28.105340 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:34.034904 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:34.034867 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:34.094561 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:34.094532 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:34.734232 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:34.734193 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:40.609949 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:40.609915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:40.669792 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:40.669753 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:40.852742 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:40.852710 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:41.145555 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:41.145514 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:41.244528 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:41.244490 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:45.021465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:45.021426 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:45.114227 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:45.114186 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:45.331874 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:45.331791 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:45.432002 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:45.431964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:48.954126 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:48.954070 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:49.599044 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:49.598999 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:49.692457 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:49.692427 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:50.022936 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:50.022902 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:55.930092 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:55.930047 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:55.990187 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:55.990131 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:38:56.280310 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.280230 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr"] Jul 24 16:38:56.280635 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.280623 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5437f9fb-b260-4cbe-91de-050a39665adf" containerName="revoke-keys" Jul 24 16:38:56.280679 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.280637 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5437f9fb-b260-4cbe-91de-050a39665adf" containerName="revoke-keys" Jul 24 16:38:56.280735 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.280708 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5437f9fb-b260-4cbe-91de-050a39665adf" containerName="revoke-keys" Jul 24 16:38:56.283067 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.283051 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" Jul 24 16:38:56.285618 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.285597 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtjsm\"" Jul 24 16:38:56.291263 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.291237 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr"] Jul 24 16:38:56.441051 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.441023 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3319f2db-cbb1-4893-8241-ec1f68a4dbd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr\" (UID: \"3319f2db-cbb1-4893-8241-ec1f68a4dbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" Jul 24 16:38:56.542086 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.541979 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3319f2db-cbb1-4893-8241-ec1f68a4dbd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr\" (UID: \"3319f2db-cbb1-4893-8241-ec1f68a4dbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" Jul 24 16:38:56.542672 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.542652 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3319f2db-cbb1-4893-8241-ec1f68a4dbd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr\" (UID: \"3319f2db-cbb1-4893-8241-ec1f68a4dbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" Jul 24 16:38:56.595848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.595812 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" Jul 24 16:38:56.736887 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.736856 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr"] Jul 24 16:38:56.738895 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:38:56.738854 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3319f2db_cbb1_4893_8241_ec1f68a4dbd2.slice/crio-c39e919c4c8466b24c10a19fb4a6c4afaa228c583fe07b993b3ccfed8f4937fb WatchSource:0}: Error finding container c39e919c4c8466b24c10a19fb4a6c4afaa228c583fe07b993b3ccfed8f4937fb: Status 404 returned error can't find the container with id c39e919c4c8466b24c10a19fb4a6c4afaa228c583fe07b993b3ccfed8f4937fb Jul 24 16:38:56.803066 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:56.802992 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" event={"ID":"3319f2db-cbb1-4893-8241-ec1f68a4dbd2","Type":"ContainerStarted","Data":"c39e919c4c8466b24c10a19fb4a6c4afaa228c583fe07b993b3ccfed8f4937fb"} Jul 24 16:38:57.809867 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:57.809767 2560 generic.go:356] "Generic (PLEG): container finished" podID="3319f2db-cbb1-4893-8241-ec1f68a4dbd2" containerID="834a761fa7837b6590ba8816e7fc22d34dc4663dad44f5d966878afad1799755" exitCode=0 Jul 24 16:38:57.809867 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:57.809835 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" event={"ID":"3319f2db-cbb1-4893-8241-ec1f68a4dbd2","Type":"ContainerDied","Data":"834a761fa7837b6590ba8816e7fc22d34dc4663dad44f5d966878afad1799755"} Jul 24 16:38:58.944946 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:58.944923 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" Jul 24 16:38:59.067448 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:59.067363 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3319f2db-cbb1-4893-8241-ec1f68a4dbd2-maas-api-service-ca\") pod \"3319f2db-cbb1-4893-8241-ec1f68a4dbd2\" (UID: \"3319f2db-cbb1-4893-8241-ec1f68a4dbd2\") " Jul 24 16:38:59.067709 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:59.067687 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3319f2db-cbb1-4893-8241-ec1f68a4dbd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3319f2db-cbb1-4893-8241-ec1f68a4dbd2" (UID: "3319f2db-cbb1-4893-8241-ec1f68a4dbd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:38:59.168187 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:59.168163 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3319f2db-cbb1-4893-8241-ec1f68a4dbd2-maas-api-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:38:59.821566 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:59.821530 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" event={"ID":"3319f2db-cbb1-4893-8241-ec1f68a4dbd2","Type":"ContainerDied","Data":"c39e919c4c8466b24c10a19fb4a6c4afaa228c583fe07b993b3ccfed8f4937fb"} Jul 24 16:38:59.821566 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:59.821561 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr" Jul 24 16:38:59.821790 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:38:59.821589 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c39e919c4c8466b24c10a19fb4a6c4afaa228c583fe07b993b3ccfed8f4937fb" Jul 24 16:39:06.580411 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:06.580381 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:06.650693 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:06.650666 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:06.712435 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:06.712404 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:11.358222 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:11.358129 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr"] Jul 24 16:39:11.361073 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:11.361048 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebaa9e95-c4xvr"] Jul 24 16:39:11.361259 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:11.361234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:11.750441 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:11.750330 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:13.166788 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:13.166756 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3319f2db-cbb1-4893-8241-ec1f68a4dbd2" path="/var/lib/kubelet/pods/3319f2db-cbb1-4893-8241-ec1f68a4dbd2/volumes" Jul 24 16:39:17.607818 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:17.607773 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:17.666687 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:17.666651 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:18.140611 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:18.140530 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:18.248059 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:18.248027 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:18.340639 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:18.340599 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:18.447658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:18.447547 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:24.205584 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:24.205549 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:24.271836 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:24.271804 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:24.353272 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:24.353236 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:24.455684 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:24.455603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:24.555240 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:24.555207 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:30.245633 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:30.245604 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:30.346972 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:30.346937 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:37.162436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:37.162396 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:37.457297 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:37.457218 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:38.001657 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:38.001610 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:40.791004 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:40.790971 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:40.885905 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:40.885859 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:41.070204 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:41.070070 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:41.176826 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:41.176793 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:41.701012 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:41.700980 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:41.774563 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:41.774535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:45.297783 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:45.297732 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:47.428471 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:47.428438 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:47.532290 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:47.532117 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:52.797365 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:52.797333 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:52.897308 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:52.897253 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:39:59.787896 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:39:59.787859 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:40:35.287599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:40:35.287569 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:40:35.384330 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:40:35.384297 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:40:35.485451 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:40:35.485406 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:40:39.881853 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:40:39.881809 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:40:43.109633 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:40:43.109367 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:40:48.066317 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:40:48.066262 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:21.130293 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:21.130266 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:41:21.136064 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:21.136041 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:41:24.801260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:24.801204 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:25.130416 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:25.130381 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:30.622558 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:30.622523 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:44.031925 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:44.031875 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:44.572787 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:44.572755 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:44.640102 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:44.640070 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:44.864890 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:44.864853 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:45.110915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:45.110880 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:46.252878 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:46.252851 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:46.356866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:46.356834 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:46.450817 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:46.450785 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:47.250918 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:47.250881 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:47.342738 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:47.342699 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:47.442522 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:47.442460 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:48.880462 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:48.880428 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:54.911881 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:54.911850 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:55.011432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:55.011396 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:41:55.606941 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:41:55.606881 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:02.064017 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:02.063985 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:02.122237 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:02.122206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:02.190270 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:02.190235 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:02.292029 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:02.291993 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:12.794784 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:12.794752 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:13.160394 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:13.160351 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:21.228339 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:21.228305 2560 scope.go:117] "RemoveContainer" containerID="76ddc6c302e8fe1b615e3d87c78c7a78552600a799ec1355089ca82531109e9a" Jul 24 16:42:34.158374 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:34.158329 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:40.177138 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:40.177092 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:40.278019 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:40.277987 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:40.552700 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:40.552607 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:40.880323 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:40.880281 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:43.141027 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:43.140988 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:43.828114 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:43.828080 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:43.929536 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:43.929511 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:43.997297 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:43.997262 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:44.097602 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:44.097517 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:44.414401 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:44.414357 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:44.522462 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:44.522413 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:48.373441 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:48.373405 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:48.776084 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:48.775999 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:49.065995 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:49.065916 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:49.125298 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:49.125266 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:51.819056 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:51.819026 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:51.901978 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:51.901951 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:51.963449 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:51.963420 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:52.282661 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:52.282628 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:52.499338 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:52.499300 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:52.594744 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:52.594665 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:42:56.320703 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:42:56.320652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:43:04.510289 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:04.510258 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:43:26.196404 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:26.196351 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:43:33.947106 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.947069 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk"] Jul 24 16:43:33.947794 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.947775 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3319f2db-cbb1-4893-8241-ec1f68a4dbd2" containerName="revoke-keys" Jul 24 16:43:33.947915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.947797 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3319f2db-cbb1-4893-8241-ec1f68a4dbd2" containerName="revoke-keys" Jul 24 16:43:33.947915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.947889 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3319f2db-cbb1-4893-8241-ec1f68a4dbd2" containerName="revoke-keys" Jul 24 16:43:33.950851 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.950832 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:33.954343 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.954320 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3881c4\"/\"openshift-service-ca.crt\"" Jul 24 16:43:33.954343 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.954335 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3881c4\"/\"kube-root-ca.crt\"" Jul 24 16:43:33.954517 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.954378 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3881c4\"/\"test-model-3881c4-kserve-self-signed-certs\"" Jul 24 16:43:33.954517 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.954379 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3881c4\"/\"default-dockercfg-8x9cp\"" Jul 24 16:43:33.959796 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.959771 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk"] Jul 24 16:43:33.975008 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:33.974969 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:43:34.089817 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.089779 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/176f16da-f462-41f3-8bc6-727428fa6c76-kube-api-access-h8lqg\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.089985 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.089846 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-model-cache\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.089985 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.089906 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176f16da-f462-41f3-8bc6-727428fa6c76-tls-certs\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.089985 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.089948 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-tmp-dir\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.089985 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.089966 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-dshm\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.090124 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.090008 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-home\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.191488 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.191452 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/176f16da-f462-41f3-8bc6-727428fa6c76-kube-api-access-h8lqg\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.191665 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.191523 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-model-cache\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.191665 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.191598 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176f16da-f462-41f3-8bc6-727428fa6c76-tls-certs\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.191665 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.191648 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-tmp-dir\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.191831 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.191675 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-dshm\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.191831 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.191705 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-home\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.191937 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.191891 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-model-cache\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.192019 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.191993 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-tmp-dir\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.192078 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.192019 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-home\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.194161 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.194123 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-dshm\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.194341 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.194326 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176f16da-f462-41f3-8bc6-727428fa6c76-tls-certs\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.199942 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.199886 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/176f16da-f462-41f3-8bc6-727428fa6c76-kube-api-access-h8lqg\") pod \"test-model-3881c4-kserve-f7c945dc9-hvgdk\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.260714 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.260678 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:34.398798 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:34.398776 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk"] Jul 24 16:43:34.401088 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:43:34.401057 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176f16da_f462_41f3_8bc6_727428fa6c76.slice/crio-a7c58506b4cb4d416f6a37ec171894a1e0771da19184b8ad9d1d9f0431187c1e WatchSource:0}: Error finding container a7c58506b4cb4d416f6a37ec171894a1e0771da19184b8ad9d1d9f0431187c1e: Status 404 returned error can't find the container with id a7c58506b4cb4d416f6a37ec171894a1e0771da19184b8ad9d1d9f0431187c1e Jul 24 16:43:35.206518 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:35.206483 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" event={"ID":"176f16da-f462-41f3-8bc6-727428fa6c76","Type":"ContainerStarted","Data":"811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011"} Jul 24 16:43:35.206878 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:35.206524 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" event={"ID":"176f16da-f462-41f3-8bc6-727428fa6c76","Type":"ContainerStarted","Data":"a7c58506b4cb4d416f6a37ec171894a1e0771da19184b8ad9d1d9f0431187c1e"} Jul 24 16:43:35.224168 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:35.224102 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" podStartSLOduration=2.224089083 podStartE2EDuration="2.224089083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:35.222331562 +0000 UTC m=+2234.653966921" watchObservedRunningTime="2026-07-24 16:43:35.224089083 +0000 UTC m=+2234.655724478" Jul 24 16:43:44.261250 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:44.261166 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:44.261250 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:44.261226 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:44.273581 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:44.273555 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:43:45.273138 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:43:45.273109 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:44:07.598199 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:07.598168 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:16.043066 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:16.043027 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:16.923692 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:16.923649 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk"] Jul 24 16:44:16.946407 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:16.946380 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:17.070455 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:17.070420 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf"] Jul 24 16:44:17.074292 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:17.074272 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" Jul 24 16:44:17.076705 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:17.076682 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtjsm\"" Jul 24 16:44:17.081719 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:17.081696 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf"] Jul 24 16:44:17.206579 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:17.206487 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8066daab-f27d-4643-b2ef-dc0f04db7ed1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf\" (UID: \"8066daab-f27d-4643-b2ef-dc0f04db7ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" Jul 24 16:44:17.307496 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:17.307461 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8066daab-f27d-4643-b2ef-dc0f04db7ed1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf\" (UID: \"8066daab-f27d-4643-b2ef-dc0f04db7ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" Jul 24 16:44:17.308354 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:17.308329 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8066daab-f27d-4643-b2ef-dc0f04db7ed1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf\" (UID: \"8066daab-f27d-4643-b2ef-dc0f04db7ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" Jul 24 16:44:17.390912 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:17.390872 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" Jul 24 16:44:17.537071 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:17.537047 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf"] Jul 24 16:44:17.538935 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:44:17.538900 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8066daab_f27d_4643_b2ef_dc0f04db7ed1.slice/crio-ec8fcad13270a16bf7dfc10022981cb7dae1c49f5d7f4c256c9484732fd11428 WatchSource:0}: Error finding container ec8fcad13270a16bf7dfc10022981cb7dae1c49f5d7f4c256c9484732fd11428: Status 404 returned error can't find the container with id ec8fcad13270a16bf7dfc10022981cb7dae1c49f5d7f4c256c9484732fd11428 Jul 24 16:44:18.435370 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:18.435339 2560 generic.go:356] "Generic (PLEG): container finished" podID="8066daab-f27d-4643-b2ef-dc0f04db7ed1" containerID="54e16c0f7217d8667877a6cda56926ca3b54922bef4262c1bc294f07fd824ae6" exitCode=0 Jul 24 16:44:18.435766 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:18.435386 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" event={"ID":"8066daab-f27d-4643-b2ef-dc0f04db7ed1","Type":"ContainerDied","Data":"54e16c0f7217d8667877a6cda56926ca3b54922bef4262c1bc294f07fd824ae6"} Jul 24 16:44:18.435766 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:18.435430 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" event={"ID":"8066daab-f27d-4643-b2ef-dc0f04db7ed1","Type":"ContainerStarted","Data":"ec8fcad13270a16bf7dfc10022981cb7dae1c49f5d7f4c256c9484732fd11428"} Jul 24 16:44:19.579991 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:19.579968 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" Jul 24 16:44:19.730875 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:19.730792 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8066daab-f27d-4643-b2ef-dc0f04db7ed1-maas-api-service-ca\") pod \"8066daab-f27d-4643-b2ef-dc0f04db7ed1\" (UID: \"8066daab-f27d-4643-b2ef-dc0f04db7ed1\") " Jul 24 16:44:19.731297 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:19.731263 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8066daab-f27d-4643-b2ef-dc0f04db7ed1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8066daab-f27d-4643-b2ef-dc0f04db7ed1" (UID: "8066daab-f27d-4643-b2ef-dc0f04db7ed1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:44:19.832195 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:19.832137 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8066daab-f27d-4643-b2ef-dc0f04db7ed1-maas-api-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:44:20.445863 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:20.445827 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" Jul 24 16:44:20.446039 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:20.445873 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf" event={"ID":"8066daab-f27d-4643-b2ef-dc0f04db7ed1","Type":"ContainerDied","Data":"ec8fcad13270a16bf7dfc10022981cb7dae1c49f5d7f4c256c9484732fd11428"} Jul 24 16:44:20.446039 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:20.445927 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec8fcad13270a16bf7dfc10022981cb7dae1c49f5d7f4c256c9484732fd11428" Jul 24 16:44:27.655369 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:27.655334 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:27.753762 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:27.753726 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:27.854457 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:27.854418 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:31.943427 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:31.943382 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" podUID="176f16da-f462-41f3-8bc6-727428fa6c76" containerName="main" containerID="cri-o://811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011" gracePeriod=45 Jul 24 16:44:32.145620 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.145560 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:32.156246 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.156194 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf"] Jul 24 16:44:32.160241 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.160215 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ef1797-fptpf"] Jul 24 16:44:32.198456 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.198407 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:44:32.257494 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.257466 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-tmp-dir\") pod \"176f16da-f462-41f3-8bc6-727428fa6c76\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " Jul 24 16:44:32.257675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.257505 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-dshm\") pod \"176f16da-f462-41f3-8bc6-727428fa6c76\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " Jul 24 16:44:32.257675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.257536 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-home\") pod \"176f16da-f462-41f3-8bc6-727428fa6c76\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " Jul 24 16:44:32.257675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.257591 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-model-cache\") pod \"176f16da-f462-41f3-8bc6-727428fa6c76\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " Jul 24 16:44:32.257675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.257628 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/176f16da-f462-41f3-8bc6-727428fa6c76-kube-api-access-h8lqg\") pod \"176f16da-f462-41f3-8bc6-727428fa6c76\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " Jul 24 16:44:32.257675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.257673 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176f16da-f462-41f3-8bc6-727428fa6c76-tls-certs\") pod \"176f16da-f462-41f3-8bc6-727428fa6c76\" (UID: \"176f16da-f462-41f3-8bc6-727428fa6c76\") " Jul 24 16:44:32.257933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.257777 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "176f16da-f462-41f3-8bc6-727428fa6c76" (UID: "176f16da-f462-41f3-8bc6-727428fa6c76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:44:32.257933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.257830 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-home" (OuterVolumeSpecName: "home") pod "176f16da-f462-41f3-8bc6-727428fa6c76" (UID: "176f16da-f462-41f3-8bc6-727428fa6c76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:44:32.257933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.257915 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-model-cache" (OuterVolumeSpecName: "model-cache") pod "176f16da-f462-41f3-8bc6-727428fa6c76" (UID: "176f16da-f462-41f3-8bc6-727428fa6c76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:44:32.258086 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.258064 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-tmp-dir\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:44:32.258086 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.258079 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-home\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:44:32.258217 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.258088 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-model-cache\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:44:32.259900 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.259873 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-dshm" (OuterVolumeSpecName: "dshm") pod "176f16da-f462-41f3-8bc6-727428fa6c76" (UID: "176f16da-f462-41f3-8bc6-727428fa6c76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:44:32.260012 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.259933 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176f16da-f462-41f3-8bc6-727428fa6c76-kube-api-access-h8lqg" (OuterVolumeSpecName: "kube-api-access-h8lqg") pod "176f16da-f462-41f3-8bc6-727428fa6c76" (UID: "176f16da-f462-41f3-8bc6-727428fa6c76"). InnerVolumeSpecName "kube-api-access-h8lqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:44:32.260012 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.259957 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176f16da-f462-41f3-8bc6-727428fa6c76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "176f16da-f462-41f3-8bc6-727428fa6c76" (UID: "176f16da-f462-41f3-8bc6-727428fa6c76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:44:32.359436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.359384 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176f16da-f462-41f3-8bc6-727428fa6c76-tls-certs\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:44:32.359436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.359420 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176f16da-f462-41f3-8bc6-727428fa6c76-dshm\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:44:32.359436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.359433 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/176f16da-f462-41f3-8bc6-727428fa6c76-kube-api-access-h8lqg\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:44:32.506389 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.506298 2560 generic.go:356] "Generic (PLEG): container finished" podID="176f16da-f462-41f3-8bc6-727428fa6c76" containerID="811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011" exitCode=0 Jul 24 16:44:32.506389 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.506350 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" Jul 24 16:44:32.506617 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.506383 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" event={"ID":"176f16da-f462-41f3-8bc6-727428fa6c76","Type":"ContainerDied","Data":"811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011"} Jul 24 16:44:32.506617 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.506426 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk" event={"ID":"176f16da-f462-41f3-8bc6-727428fa6c76","Type":"ContainerDied","Data":"a7c58506b4cb4d416f6a37ec171894a1e0771da19184b8ad9d1d9f0431187c1e"} Jul 24 16:44:32.506617 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.506442 2560 scope.go:117] "RemoveContainer" containerID="811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011" Jul 24 16:44:32.516355 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.516337 2560 scope.go:117] "RemoveContainer" containerID="811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011" Jul 24 16:44:32.516603 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:44:32.516583 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011\": container with ID starting with 811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011 not found: ID does not exist" containerID="811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011" Jul 24 16:44:32.516666 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.516615 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011"} err="failed to get container status \"811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011\": rpc error: code = NotFound desc = could not find container \"811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011\": container with ID starting with 811356e78d113544461703354891bf9a4482baa25861a788d721947d78a70011 not found: ID does not exist" Jul 24 16:44:32.529639 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.529615 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk"] Jul 24 16:44:32.534852 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.534832 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3881c4/test-model-3881c4-kserve-f7c945dc9-hvgdk"] Jul 24 16:44:32.560789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:32.560755 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:33.166428 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:33.166394 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="176f16da-f462-41f3-8bc6-727428fa6c76" path="/var/lib/kubelet/pods/176f16da-f462-41f3-8bc6-727428fa6c76/volumes" Jul 24 16:44:33.166772 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:33.166695 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8066daab-f27d-4643-b2ef-dc0f04db7ed1" path="/var/lib/kubelet/pods/8066daab-f27d-4643-b2ef-dc0f04db7ed1/volumes" Jul 24 16:44:38.300210 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:38.300172 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:38.366723 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:38.366694 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:38.564803 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:38.564716 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:38.793043 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:38.793008 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:41.392831 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:41.392795 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:49.478764 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:49.478732 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:49.578937 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:49.578909 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:49.643232 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:49.643199 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:54.002407 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:54.002377 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:44:54.371189 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:44:54.371138 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:00.662735 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:00.662706 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:00.763067 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:00.763030 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:00.941426 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:00.941343 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:01.148814 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:01.148778 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:01.478546 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:01.478517 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:01.777870 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:01.777786 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:02.120717 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:02.120691 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:04.566216 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:04.566185 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:04.659334 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:04.659300 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:04.745744 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:04.745714 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:04.845431 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:04.845347 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:05.353816 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:05.353779 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:05.449888 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:05.449849 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:09.042097 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:09.042064 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:09.510362 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:09.510330 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:09.890532 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:09.890491 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:09.986476 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:09.986440 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:12.597101 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:12.597023 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:12.677399 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:12.677361 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:12.740399 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:12.740366 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:13.004685 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:13.004653 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:13.369379 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:13.369352 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:13.453071 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:13.453040 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:17.091321 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:17.091262 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:21.263770 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:21.263740 2560 scope.go:117] "RemoveContainer" containerID="834a761fa7837b6590ba8816e7fc22d34dc4663dad44f5d966878afad1799755" Jul 24 16:45:25.113361 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.113322 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv"] Jul 24 16:45:25.113745 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.113717 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8066daab-f27d-4643-b2ef-dc0f04db7ed1" containerName="revoke-keys" Jul 24 16:45:25.113745 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.113728 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8066daab-f27d-4643-b2ef-dc0f04db7ed1" containerName="revoke-keys" Jul 24 16:45:25.113834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.113752 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176f16da-f462-41f3-8bc6-727428fa6c76" containerName="main" Jul 24 16:45:25.113834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.113758 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="176f16da-f462-41f3-8bc6-727428fa6c76" containerName="main" Jul 24 16:45:25.113834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.113822 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8066daab-f27d-4643-b2ef-dc0f04db7ed1" containerName="revoke-keys" Jul 24 16:45:25.113834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.113829 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="176f16da-f462-41f3-8bc6-727428fa6c76" containerName="main" Jul 24 16:45:25.116801 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.116786 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" Jul 24 16:45:25.119200 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.119175 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtjsm\"" Jul 24 16:45:25.123531 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.123509 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv"] Jul 24 16:45:25.273888 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.273854 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7580c-7614-481e-8a5c-6e1cbde6a052-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv\" (UID: \"5ac7580c-7614-481e-8a5c-6e1cbde6a052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" Jul 24 16:45:25.375413 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.375332 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7580c-7614-481e-8a5c-6e1cbde6a052-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv\" (UID: \"5ac7580c-7614-481e-8a5c-6e1cbde6a052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" Jul 24 16:45:25.375933 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.375912 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7580c-7614-481e-8a5c-6e1cbde6a052-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv\" (UID: \"5ac7580c-7614-481e-8a5c-6e1cbde6a052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" Jul 24 16:45:25.430408 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.430372 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" Jul 24 16:45:25.565760 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.565732 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv"] Jul 24 16:45:25.567969 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:45:25.567939 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac7580c_7614_481e_8a5c_6e1cbde6a052.slice/crio-769b4679f736ab62726f9245079bd4dc8815817ab5fae9775374c7f94d752465 WatchSource:0}: Error finding container 769b4679f736ab62726f9245079bd4dc8815817ab5fae9775374c7f94d752465: Status 404 returned error can't find the container with id 769b4679f736ab62726f9245079bd4dc8815817ab5fae9775374c7f94d752465 Jul 24 16:45:25.781341 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:25.781308 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" event={"ID":"5ac7580c-7614-481e-8a5c-6e1cbde6a052","Type":"ContainerStarted","Data":"769b4679f736ab62726f9245079bd4dc8815817ab5fae9775374c7f94d752465"} Jul 24 16:45:26.787458 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:26.787425 2560 generic.go:356] "Generic (PLEG): container finished" podID="5ac7580c-7614-481e-8a5c-6e1cbde6a052" containerID="369b4011c0ba3dc115b562cf8416aa961684b82c25472322d7ad7cf03e801706" exitCode=0 Jul 24 16:45:26.787850 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:26.787488 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" event={"ID":"5ac7580c-7614-481e-8a5c-6e1cbde6a052","Type":"ContainerDied","Data":"369b4011c0ba3dc115b562cf8416aa961684b82c25472322d7ad7cf03e801706"} Jul 24 16:45:27.927642 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:27.927621 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" Jul 24 16:45:28.099809 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:28.099728 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7580c-7614-481e-8a5c-6e1cbde6a052-maas-api-service-ca\") pod \"5ac7580c-7614-481e-8a5c-6e1cbde6a052\" (UID: \"5ac7580c-7614-481e-8a5c-6e1cbde6a052\") " Jul 24 16:45:28.100116 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:28.100090 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac7580c-7614-481e-8a5c-6e1cbde6a052-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ac7580c-7614-481e-8a5c-6e1cbde6a052" (UID: "5ac7580c-7614-481e-8a5c-6e1cbde6a052"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:45:28.201222 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:28.201191 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7580c-7614-481e-8a5c-6e1cbde6a052-maas-api-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:45:28.800705 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:28.800676 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" Jul 24 16:45:28.800875 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:28.800715 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv" event={"ID":"5ac7580c-7614-481e-8a5c-6e1cbde6a052","Type":"ContainerDied","Data":"769b4679f736ab62726f9245079bd4dc8815817ab5fae9775374c7f94d752465"} Jul 24 16:45:28.800875 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:28.800785 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="769b4679f736ab62726f9245079bd4dc8815817ab5fae9775374c7f94d752465" Jul 24 16:45:40.805064 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:40.805032 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:40.906012 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:40.905968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:41.006267 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:41.006234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:45.196899 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:45.196863 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv"] Jul 24 16:45:45.200303 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:45.200273 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:45.202253 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:45.202226 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aac1d1-zqpmv"] Jul 24 16:45:45.578513 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:45.578435 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:47.166735 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:47.166704 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac7580c-7614-481e-8a5c-6e1cbde6a052" path="/var/lib/kubelet/pods/5ac7580c-7614-481e-8a5c-6e1cbde6a052/volumes" Jul 24 16:45:51.238912 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:51.238876 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:51.339859 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:51.339822 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:51.501878 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:51.501846 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:45:51.726243 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:45:51.726207 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:07.334940 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:07.334903 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:07.436006 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:07.435973 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:07.535895 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:07.535869 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:11.932280 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:11.932243 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:12.310097 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:12.310023 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:18.098130 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:18.098097 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:18.164115 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:18.164086 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:18.362206 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:18.362176 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:18.588863 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:18.588828 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:18.960777 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:18.960730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:19.265245 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:19.265124 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:19.616368 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:19.616335 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:21.162892 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:21.162864 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:46:21.170373 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:21.170351 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:46:22.003307 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:22.003272 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:22.080245 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:22.080196 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:22.147671 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:22.147633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:22.430062 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:22.430027 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:22.808561 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:22.808487 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:22.881318 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:22.881283 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:26.497001 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:26.496963 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:26.828985 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:26.828899 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:27.145767 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:27.145730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:27.270842 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:27.270813 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:30.030473 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:30.030428 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:30.136660 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:30.136628 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:30.307392 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:30.307311 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:30.547836 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:30.547803 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:30.839916 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:30.839884 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:30.924170 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:30.924113 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:34.530512 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:34.530478 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:41.728842 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:41.728725 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:41.829720 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:41.829685 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:46:50.888080 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:46:50.888040 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:47:20.649816 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.649783 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2"] Jul 24 16:47:20.651981 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.650210 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac7580c-7614-481e-8a5c-6e1cbde6a052" containerName="revoke-keys" Jul 24 16:47:20.651981 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.650222 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac7580c-7614-481e-8a5c-6e1cbde6a052" containerName="revoke-keys" Jul 24 16:47:20.651981 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.650305 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac7580c-7614-481e-8a5c-6e1cbde6a052" containerName="revoke-keys" Jul 24 16:47:20.653230 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.653214 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.656658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.656629 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"default-dockercfg-8mj7j\"" Jul 24 16:47:20.656807 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.656629 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"openshift-service-ca.crt\"" Jul 24 16:47:20.656807 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.656629 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"auth-test-model-a29da6-kserve-self-signed-certs\"" Jul 24 16:47:20.656807 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.656636 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"kube-root-ca.crt\"" Jul 24 16:47:20.663460 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.663435 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2"] Jul 24 16:47:20.685540 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.685508 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:47:20.780042 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.780012 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-dshm\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.780216 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.780074 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-model-cache\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.780216 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.780116 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-tmp-dir\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.780299 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.780215 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z6h8\" (UniqueName: \"kubernetes.io/projected/846863ef-7b07-4453-81c5-8a7f1cec80ad-kube-api-access-8z6h8\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.780299 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.780238 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846863ef-7b07-4453-81c5-8a7f1cec80ad-tls-certs\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.780299 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.780276 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-home\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.880837 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.880807 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-model-cache\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.880837 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.880837 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-tmp-dir\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.881043 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.880869 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6h8\" (UniqueName: \"kubernetes.io/projected/846863ef-7b07-4453-81c5-8a7f1cec80ad-kube-api-access-8z6h8\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.881043 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.880888 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846863ef-7b07-4453-81c5-8a7f1cec80ad-tls-certs\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.881043 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.881008 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-home\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.881270 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.881098 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-dshm\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.881270 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.881248 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-model-cache\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.881385 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.881307 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-tmp-dir\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.881385 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.881320 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-home\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.883362 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.883337 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-dshm\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.883552 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.883536 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846863ef-7b07-4453-81c5-8a7f1cec80ad-tls-certs\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.889120 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.889097 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6h8\" (UniqueName: \"kubernetes.io/projected/846863ef-7b07-4453-81c5-8a7f1cec80ad-kube-api-access-8z6h8\") pod \"auth-test-model-a29da6-kserve-7445d94fb7-j6qk2\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:20.966617 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.966549 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"default-dockercfg-8mj7j\"" Jul 24 16:47:20.974542 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:20.974523 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:21.106168 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:21.106126 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2"] Jul 24 16:47:21.107597 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:47:21.107572 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod846863ef_7b07_4453_81c5_8a7f1cec80ad.slice/crio-bc0369849c77ba6d82c7e08967cbf11a15158b4a15eaf06e0b037c50ff4b4bbe WatchSource:0}: Error finding container bc0369849c77ba6d82c7e08967cbf11a15158b4a15eaf06e0b037c50ff4b4bbe: Status 404 returned error can't find the container with id bc0369849c77ba6d82c7e08967cbf11a15158b4a15eaf06e0b037c50ff4b4bbe Jul 24 16:47:21.375152 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:21.375116 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" event={"ID":"846863ef-7b07-4453-81c5-8a7f1cec80ad","Type":"ContainerStarted","Data":"46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74"} Jul 24 16:47:21.375342 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:21.375177 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" event={"ID":"846863ef-7b07-4453-81c5-8a7f1cec80ad","Type":"ContainerStarted","Data":"bc0369849c77ba6d82c7e08967cbf11a15158b4a15eaf06e0b037c50ff4b4bbe"} Jul 24 16:47:21.391863 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:21.391821 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" podStartSLOduration=1.391805516 podStartE2EDuration="1.391805516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:21.391286366 +0000 UTC m=+2460.822921727" watchObservedRunningTime="2026-07-24 16:47:21.391805516 +0000 UTC m=+2460.823440878" Jul 24 16:47:30.975244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:30.975191 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:30.975244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:30.975262 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:30.987802 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:30.987774 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:47:31.435450 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:47:31.435423 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:48:27.582670 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:48:27.582637 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:48:40.627688 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:48:40.627656 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:49:15.437104 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:15.437071 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:49:32.082631 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:32.082594 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:49:33.264314 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:33.264278 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2"] Jul 24 16:49:33.281440 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:33.281397 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:49:48.233066 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.233012 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:49:48.285079 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.285034 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" podUID="846863ef-7b07-4453-81c5-8a7f1cec80ad" containerName="main" containerID="cri-o://46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74" gracePeriod=45 Jul 24 16:49:48.545169 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.545127 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:49:48.606241 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606218 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-dshm\") pod \"846863ef-7b07-4453-81c5-8a7f1cec80ad\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " Jul 24 16:49:48.606415 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606277 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z6h8\" (UniqueName: \"kubernetes.io/projected/846863ef-7b07-4453-81c5-8a7f1cec80ad-kube-api-access-8z6h8\") pod \"846863ef-7b07-4453-81c5-8a7f1cec80ad\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " Jul 24 16:49:48.606415 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606315 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-home\") pod \"846863ef-7b07-4453-81c5-8a7f1cec80ad\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " Jul 24 16:49:48.606415 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606376 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-model-cache\") pod \"846863ef-7b07-4453-81c5-8a7f1cec80ad\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " Jul 24 16:49:48.606415 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606403 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-tmp-dir\") pod \"846863ef-7b07-4453-81c5-8a7f1cec80ad\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " Jul 24 16:49:48.606612 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606443 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846863ef-7b07-4453-81c5-8a7f1cec80ad-tls-certs\") pod \"846863ef-7b07-4453-81c5-8a7f1cec80ad\" (UID: \"846863ef-7b07-4453-81c5-8a7f1cec80ad\") " Jul 24 16:49:48.606670 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606645 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-home" (OuterVolumeSpecName: "home") pod "846863ef-7b07-4453-81c5-8a7f1cec80ad" (UID: "846863ef-7b07-4453-81c5-8a7f1cec80ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:49:48.606714 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606662 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "846863ef-7b07-4453-81c5-8a7f1cec80ad" (UID: "846863ef-7b07-4453-81c5-8a7f1cec80ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:49:48.606758 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606716 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "846863ef-7b07-4453-81c5-8a7f1cec80ad" (UID: "846863ef-7b07-4453-81c5-8a7f1cec80ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:49:48.606858 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606839 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-home\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:49:48.606927 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606865 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-model-cache\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:49:48.606927 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.606877 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-tmp-dir\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:49:48.608515 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.608495 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-dshm" (OuterVolumeSpecName: "dshm") pod "846863ef-7b07-4453-81c5-8a7f1cec80ad" (UID: "846863ef-7b07-4453-81c5-8a7f1cec80ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:49:48.608599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.608579 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846863ef-7b07-4453-81c5-8a7f1cec80ad-kube-api-access-8z6h8" (OuterVolumeSpecName: "kube-api-access-8z6h8") pod "846863ef-7b07-4453-81c5-8a7f1cec80ad" (UID: "846863ef-7b07-4453-81c5-8a7f1cec80ad"). InnerVolumeSpecName "kube-api-access-8z6h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:49:48.608675 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.608657 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846863ef-7b07-4453-81c5-8a7f1cec80ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "846863ef-7b07-4453-81c5-8a7f1cec80ad" (UID: "846863ef-7b07-4453-81c5-8a7f1cec80ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:49:48.707522 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.707489 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846863ef-7b07-4453-81c5-8a7f1cec80ad-tls-certs\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:49:48.707522 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.707515 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846863ef-7b07-4453-81c5-8a7f1cec80ad-dshm\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:49:48.707522 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:48.707525 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z6h8\" (UniqueName: \"kubernetes.io/projected/846863ef-7b07-4453-81c5-8a7f1cec80ad-kube-api-access-8z6h8\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:49:49.142491 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:49.142451 2560 generic.go:356] "Generic (PLEG): container finished" podID="846863ef-7b07-4453-81c5-8a7f1cec80ad" containerID="46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74" exitCode=0 Jul 24 16:49:49.142656 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:49.142502 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" Jul 24 16:49:49.142656 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:49.142520 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" event={"ID":"846863ef-7b07-4453-81c5-8a7f1cec80ad","Type":"ContainerDied","Data":"46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74"} Jul 24 16:49:49.142656 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:49.142561 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2" event={"ID":"846863ef-7b07-4453-81c5-8a7f1cec80ad","Type":"ContainerDied","Data":"bc0369849c77ba6d82c7e08967cbf11a15158b4a15eaf06e0b037c50ff4b4bbe"} Jul 24 16:49:49.142656 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:49.142583 2560 scope.go:117] "RemoveContainer" containerID="46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74" Jul 24 16:49:49.152597 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:49.152578 2560 scope.go:117] "RemoveContainer" containerID="46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74" Jul 24 16:49:49.152871 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:49:49.152849 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74\": container with ID starting with 46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74 not found: ID does not exist" containerID="46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74" Jul 24 16:49:49.152931 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:49.152880 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74"} err="failed to get container status \"46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74\": rpc error: code = NotFound desc = could not find container \"46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74\": container with ID starting with 46c67087b369cac656dcc31f9a0600905abdea8dc6e8d0fdf49e02687ea6cf74 not found: ID does not exist" Jul 24 16:49:49.167066 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:49.167041 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2"] Jul 24 16:49:49.167350 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:49.167331 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/auth-test-model-a29da6-kserve-7445d94fb7-j6qk2"] Jul 24 16:49:51.167176 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:49:51.167125 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="846863ef-7b07-4453-81c5-8a7f1cec80ad" path="/var/lib/kubelet/pods/846863ef-7b07-4453-81c5-8a7f1cec80ad/volumes" Jul 24 16:50:11.032478 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.032441 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr"] Jul 24 16:50:11.032867 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.032838 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846863ef-7b07-4453-81c5-8a7f1cec80ad" containerName="main" Jul 24 16:50:11.032867 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.032851 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="846863ef-7b07-4453-81c5-8a7f1cec80ad" containerName="main" Jul 24 16:50:11.032974 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.032962 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="846863ef-7b07-4453-81c5-8a7f1cec80ad" containerName="main" Jul 24 16:50:11.036190 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.036167 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.040077 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.040045 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"kube-root-ca.crt\"" Jul 24 16:50:11.041224 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.041200 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"sub-test-model-a29da6-kserve-self-signed-certs\"" Jul 24 16:50:11.041224 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.041213 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"openshift-service-ca.crt\"" Jul 24 16:50:11.041491 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.041204 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"default-dockercfg-8mj7j\"" Jul 24 16:50:11.047698 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.047673 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr"] Jul 24 16:50:11.078422 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.078393 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:50:11.115324 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.115272 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-model-cache\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.115324 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.115332 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-tmp-dir\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.115562 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.115356 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-dshm\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.115562 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.115387 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg5q\" (UniqueName: \"kubernetes.io/projected/26c674c6-2c51-4301-b973-cb74dc0f790c-kube-api-access-shg5q\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.115562 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.115469 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c674c6-2c51-4301-b973-cb74dc0f790c-tls-certs\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.115562 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.115501 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-home\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.216071 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.216015 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c674c6-2c51-4301-b973-cb74dc0f790c-tls-certs\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.216071 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.216075 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-home\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.216317 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.216178 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-model-cache\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.216317 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.216226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-tmp-dir\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.216317 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.216262 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-dshm\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.216317 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.216308 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg5q\" (UniqueName: \"kubernetes.io/projected/26c674c6-2c51-4301-b973-cb74dc0f790c-kube-api-access-shg5q\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.216557 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.216533 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-home\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.216632 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.216580 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-model-cache\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.216693 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.216659 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-tmp-dir\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.218533 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.218508 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-dshm\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.218848 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.218831 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c674c6-2c51-4301-b973-cb74dc0f790c-tls-certs\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.224601 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.224579 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg5q\" (UniqueName: \"kubernetes.io/projected/26c674c6-2c51-4301-b973-cb74dc0f790c-kube-api-access-shg5q\") pod \"sub-test-model-a29da6-kserve-55d59d5477-f6wlr\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.347179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.347074 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:11.484098 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:11.484073 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr"] Jul 24 16:50:11.486769 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:50:11.486743 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26c674c6_2c51_4301_b973_cb74dc0f790c.slice/crio-3e6974323136310af36cc12d7f1fb5d5e02ced2461dff8019a19a52745957673 WatchSource:0}: Error finding container 3e6974323136310af36cc12d7f1fb5d5e02ced2461dff8019a19a52745957673: Status 404 returned error can't find the container with id 3e6974323136310af36cc12d7f1fb5d5e02ced2461dff8019a19a52745957673 Jul 24 16:50:12.260617 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:12.260579 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" event={"ID":"26c674c6-2c51-4301-b973-cb74dc0f790c","Type":"ContainerStarted","Data":"245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e"} Jul 24 16:50:12.261016 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:12.260625 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" event={"ID":"26c674c6-2c51-4301-b973-cb74dc0f790c","Type":"ContainerStarted","Data":"3e6974323136310af36cc12d7f1fb5d5e02ced2461dff8019a19a52745957673"} Jul 24 16:50:12.279118 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:12.279066 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" podStartSLOduration=1.279052515 podStartE2EDuration="1.279052515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:50:12.277163919 +0000 UTC m=+2631.708799272" watchObservedRunningTime="2026-07-24 16:50:12.279052515 +0000 UTC m=+2631.710687874" Jul 24 16:50:21.318399 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:21.318368 2560 scope.go:117] "RemoveContainer" containerID="54e16c0f7217d8667877a6cda56926ca3b54922bef4262c1bc294f07fd824ae6" Jul 24 16:50:21.347289 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:21.347268 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:21.347387 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:21.347313 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:21.360200 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:21.360179 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:22.324216 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:22.324190 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:50:54.791076 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:54.791039 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:50:57.783250 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:57.783212 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:50:58.665578 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:58.665548 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr"] Jul 24 16:50:58.685974 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:50:58.685930 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:51:01.682970 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:01.682935 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:51:13.685312 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:13.685201 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" podUID="26c674c6-2c51-4301-b973-cb74dc0f790c" containerName="main" containerID="cri-o://245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e" gracePeriod=45 Jul 24 16:51:13.940681 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:13.940626 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:51:14.023405 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023370 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-tmp-dir\") pod \"26c674c6-2c51-4301-b973-cb74dc0f790c\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " Jul 24 16:51:14.023405 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023409 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-model-cache\") pod \"26c674c6-2c51-4301-b973-cb74dc0f790c\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " Jul 24 16:51:14.023658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023445 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-home\") pod \"26c674c6-2c51-4301-b973-cb74dc0f790c\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " Jul 24 16:51:14.023658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023470 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-dshm\") pod \"26c674c6-2c51-4301-b973-cb74dc0f790c\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " Jul 24 16:51:14.023658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023551 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c674c6-2c51-4301-b973-cb74dc0f790c-tls-certs\") pod \"26c674c6-2c51-4301-b973-cb74dc0f790c\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " Jul 24 16:51:14.023658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023568 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shg5q\" (UniqueName: \"kubernetes.io/projected/26c674c6-2c51-4301-b973-cb74dc0f790c-kube-api-access-shg5q\") pod \"26c674c6-2c51-4301-b973-cb74dc0f790c\" (UID: \"26c674c6-2c51-4301-b973-cb74dc0f790c\") " Jul 24 16:51:14.023658 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023637 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-model-cache" (OuterVolumeSpecName: "model-cache") pod "26c674c6-2c51-4301-b973-cb74dc0f790c" (UID: "26c674c6-2c51-4301-b973-cb74dc0f790c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:51:14.023884 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023701 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26c674c6-2c51-4301-b973-cb74dc0f790c" (UID: "26c674c6-2c51-4301-b973-cb74dc0f790c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:51:14.023884 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023758 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-home" (OuterVolumeSpecName: "home") pod "26c674c6-2c51-4301-b973-cb74dc0f790c" (UID: "26c674c6-2c51-4301-b973-cb74dc0f790c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:51:14.023958 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023931 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-tmp-dir\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:51:14.023958 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023951 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-model-cache\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:51:14.024024 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.023966 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-home\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:51:14.025866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.025839 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c674c6-2c51-4301-b973-cb74dc0f790c-kube-api-access-shg5q" (OuterVolumeSpecName: "kube-api-access-shg5q") pod "26c674c6-2c51-4301-b973-cb74dc0f790c" (UID: "26c674c6-2c51-4301-b973-cb74dc0f790c"). InnerVolumeSpecName "kube-api-access-shg5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:51:14.025866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.025853 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-dshm" (OuterVolumeSpecName: "dshm") pod "26c674c6-2c51-4301-b973-cb74dc0f790c" (UID: "26c674c6-2c51-4301-b973-cb74dc0f790c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:51:14.026045 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.026012 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c674c6-2c51-4301-b973-cb74dc0f790c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26c674c6-2c51-4301-b973-cb74dc0f790c" (UID: "26c674c6-2c51-4301-b973-cb74dc0f790c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:51:14.124756 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.124723 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c674c6-2c51-4301-b973-cb74dc0f790c-tls-certs\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:51:14.124756 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.124753 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shg5q\" (UniqueName: \"kubernetes.io/projected/26c674c6-2c51-4301-b973-cb74dc0f790c-kube-api-access-shg5q\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:51:14.124756 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.124764 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c674c6-2c51-4301-b973-cb74dc0f790c-dshm\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:51:14.577231 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.577196 2560 generic.go:356] "Generic (PLEG): container finished" podID="26c674c6-2c51-4301-b973-cb74dc0f790c" containerID="245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e" exitCode=0 Jul 24 16:51:14.577418 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.577240 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" Jul 24 16:51:14.577418 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.577286 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" event={"ID":"26c674c6-2c51-4301-b973-cb74dc0f790c","Type":"ContainerDied","Data":"245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e"} Jul 24 16:51:14.577418 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.577324 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr" event={"ID":"26c674c6-2c51-4301-b973-cb74dc0f790c","Type":"ContainerDied","Data":"3e6974323136310af36cc12d7f1fb5d5e02ced2461dff8019a19a52745957673"} Jul 24 16:51:14.577418 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.577343 2560 scope.go:117] "RemoveContainer" containerID="245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e" Jul 24 16:51:14.586483 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.586465 2560 scope.go:117] "RemoveContainer" containerID="245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e" Jul 24 16:51:14.586735 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:51:14.586719 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e\": container with ID starting with 245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e not found: ID does not exist" containerID="245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e" Jul 24 16:51:14.586784 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.586743 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e"} err="failed to get container status \"245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e\": rpc error: code = NotFound desc = could not find container \"245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e\": container with ID starting with 245b7e88589264326bceeeb9c8cae19269ff504a8975ad4b34d717a11e55f34e not found: ID does not exist" Jul 24 16:51:14.598792 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.598765 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr"] Jul 24 16:51:14.604367 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:14.604347 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/sub-test-model-a29da6-kserve-55d59d5477-f6wlr"] Jul 24 16:51:15.167377 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:15.167342 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c674c6-2c51-4301-b973-cb74dc0f790c" path="/var/lib/kubelet/pods/26c674c6-2c51-4301-b973-cb74dc0f790c/volumes" Jul 24 16:51:21.202027 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:21.201993 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:51:21.209333 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:21.209313 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:51:27.059404 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.059371 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8"] Jul 24 16:51:27.059847 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.059786 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c674c6-2c51-4301-b973-cb74dc0f790c" containerName="main" Jul 24 16:51:27.059847 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.059798 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c674c6-2c51-4301-b973-cb74dc0f790c" containerName="main" Jul 24 16:51:27.059963 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.059877 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c674c6-2c51-4301-b973-cb74dc0f790c" containerName="main" Jul 24 16:51:27.064190 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.064171 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.068015 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.067990 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"rate-test-model-a29da6-kserve-self-signed-certs\"" Jul 24 16:51:27.068276 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.068252 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"kube-root-ca.crt\"" Jul 24 16:51:27.068400 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.068346 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"openshift-service-ca.crt\"" Jul 24 16:51:27.068469 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.068422 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a29da6\"/\"default-dockercfg-8mj7j\"" Jul 24 16:51:27.076257 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.076233 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8"] Jul 24 16:51:27.089301 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.089276 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:51:27.153718 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.153674 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tmp-dir\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.153914 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.153734 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-home\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.153914 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.153760 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-dshm\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.153914 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.153792 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5p72\" (UniqueName: \"kubernetes.io/projected/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-kube-api-access-l5p72\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.153914 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.153813 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tls-certs\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.154104 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.153945 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-model-cache\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.254645 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.254611 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tls-certs\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.254867 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.254701 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-model-cache\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.254867 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.254738 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tmp-dir\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.254983 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.254885 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-home\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.254983 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.254925 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-dshm\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.254983 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.254966 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5p72\" (UniqueName: \"kubernetes.io/projected/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-kube-api-access-l5p72\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.255164 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.255098 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tmp-dir\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.255226 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.255198 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-model-cache\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.255276 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.255268 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-home\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.257207 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.257185 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-dshm\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.257593 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.257574 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tls-certs\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.264603 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.264579 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5p72\" (UniqueName: \"kubernetes.io/projected/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-kube-api-access-l5p72\") pod \"rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.375746 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.375710 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:27.507317 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.507290 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8"] Jul 24 16:51:27.510249 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:51:27.509459 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f6beb0a_3f62_4c67_b79a_1eb5ba04d063.slice/crio-0b101b60fadd420865cae6979fa37d1114a0d04c548b5b830ecc9e2190553385 WatchSource:0}: Error finding container 0b101b60fadd420865cae6979fa37d1114a0d04c548b5b830ecc9e2190553385: Status 404 returned error can't find the container with id 0b101b60fadd420865cae6979fa37d1114a0d04c548b5b830ecc9e2190553385 Jul 24 16:51:27.646610 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.646529 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" event={"ID":"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063","Type":"ContainerStarted","Data":"868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7"} Jul 24 16:51:27.646610 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.646574 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" event={"ID":"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063","Type":"ContainerStarted","Data":"0b101b60fadd420865cae6979fa37d1114a0d04c548b5b830ecc9e2190553385"} Jul 24 16:51:27.664385 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:27.664337 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" podStartSLOduration=0.664321935 podStartE2EDuration="664.321935ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:51:27.662577855 +0000 UTC m=+2707.094213217" watchObservedRunningTime="2026-07-24 16:51:27.664321935 +0000 UTC m=+2707.095957295" Jul 24 16:51:37.376238 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:37.376205 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:37.376629 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:37.376263 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:37.389154 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:37.389118 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:51:37.710119 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:51:37.710019 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:52:11.719891 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:11.719848 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:12.635684 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:12.635649 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8"] Jul 24 16:52:12.648731 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:12.648699 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:21.354279 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:21.354248 2560 scope.go:117] "RemoveContainer" containerID="369b4011c0ba3dc115b562cf8416aa961684b82c25472322d7ad7cf03e801706" Jul 24 16:52:27.655545 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.655506 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" podUID="9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" containerName="main" containerID="cri-o://868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7" gracePeriod=45 Jul 24 16:52:27.697490 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.697447 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" podUID="9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused" Jul 24 16:52:27.912432 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.912362 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:52:27.953527 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.953491 2560 generic.go:356] "Generic (PLEG): container finished" podID="9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" containerID="868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7" exitCode=0 Jul 24 16:52:27.953683 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.953536 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" Jul 24 16:52:27.953683 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.953581 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" event={"ID":"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063","Type":"ContainerDied","Data":"868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7"} Jul 24 16:52:27.953683 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.953623 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8" event={"ID":"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063","Type":"ContainerDied","Data":"0b101b60fadd420865cae6979fa37d1114a0d04c548b5b830ecc9e2190553385"} Jul 24 16:52:27.953683 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.953644 2560 scope.go:117] "RemoveContainer" containerID="868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7" Jul 24 16:52:27.958501 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.958479 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tls-certs\") pod \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " Jul 24 16:52:27.958608 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.958569 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-model-cache\") pod \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " Jul 24 16:52:27.958659 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.958645 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-home\") pod \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " Jul 24 16:52:27.958703 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.958682 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tmp-dir\") pod \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " Jul 24 16:52:27.958777 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.958750 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5p72\" (UniqueName: \"kubernetes.io/projected/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-kube-api-access-l5p72\") pod \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " Jul 24 16:52:27.958908 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.958789 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-dshm\") pod \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\" (UID: \"9f6beb0a-3f62-4c67-b79a-1eb5ba04d063\") " Jul 24 16:52:27.958989 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.958781 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" (UID: "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:52:27.958989 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.958937 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-home" (OuterVolumeSpecName: "home") pod "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" (UID: "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:52:27.959112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.959001 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" (UID: "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:52:27.959189 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.959111 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tmp-dir\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:52:27.959189 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.959131 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-model-cache\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:52:27.959189 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.959167 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-home\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:52:27.960747 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.960721 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" (UID: "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:52:27.960850 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.960826 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-dshm" (OuterVolumeSpecName: "dshm") pod "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" (UID: "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:52:27.961074 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.961052 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-kube-api-access-l5p72" (OuterVolumeSpecName: "kube-api-access-l5p72") pod "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" (UID: "9f6beb0a-3f62-4c67-b79a-1eb5ba04d063"). InnerVolumeSpecName "kube-api-access-l5p72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:52:27.963474 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.963458 2560 scope.go:117] "RemoveContainer" containerID="868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7" Jul 24 16:52:27.963757 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:52:27.963726 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7\": container with ID starting with 868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7 not found: ID does not exist" containerID="868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7" Jul 24 16:52:27.963799 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:27.963767 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7"} err="failed to get container status \"868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7\": rpc error: code = NotFound desc = could not find container \"868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7\": container with ID starting with 868aa0e4fa8e8a07991ab839597b32d0cf4b744e8685fae286a8674faa145ef7 not found: ID does not exist" Jul 24 16:52:28.060567 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:28.060530 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5p72\" (UniqueName: \"kubernetes.io/projected/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-kube-api-access-l5p72\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:52:28.060567 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:28.060560 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-dshm\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:52:28.060757 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:28.060576 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063-tls-certs\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:52:28.275040 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:28.275012 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8"] Jul 24 16:52:28.280960 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:28.280933 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a29da6/rate-test-model-a29da6-kserve-7dd47d9f67-g5vc8"] Jul 24 16:52:29.166708 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:29.166676 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" path="/var/lib/kubelet/pods/9f6beb0a-3f62-4c67-b79a-1eb5ba04d063/volumes" Jul 24 16:52:31.810914 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:31.810882 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:32.109711 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:32.109683 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:32.291751 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:32.291720 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:35.079259 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:35.079135 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:35.168183 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:35.168124 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:35.267581 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:35.267545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:35.509984 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:35.509937 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:35.810131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:35.810034 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:35.910071 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:35.910041 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:39.562648 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:39.562615 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:50.476796 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:50.476760 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:50.778819 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:50.778739 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:50.957772 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:50.957734 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:53.773180 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:53.773120 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:53.848018 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:53.847989 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:54.220578 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:54.220543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:54.312884 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:54.312850 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:54.474629 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:54.474546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:54.571622 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:54.571590 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:55.021170 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:55.021113 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:55.119872 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:55.119836 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:52:58.728182 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:52:58.728138 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:53:26.045597 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:26.045558 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9l5g2_6a9cb2fb-a288-44d2-b490-f90c66e90120/payload-processing/0.log" Jul 24 16:53:31.334743 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.334713 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd"] Jul 24 16:53:31.335170 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.335135 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" containerName="main" Jul 24 16:53:31.335244 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.335173 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" containerName="main" Jul 24 16:53:31.335292 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.335260 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6beb0a-3f62-4c67-b79a-1eb5ba04d063" containerName="main" Jul 24 16:53:31.338318 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.338301 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.340740 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.340713 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b04709\"/\"kube-root-ca.crt\"" Jul 24 16:53:31.341883 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.341711 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b04709\"/\"openshift-service-ca.crt\"" Jul 24 16:53:31.341883 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.341826 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b04709\"/\"default-dockercfg-m5prm\"" Jul 24 16:53:31.342229 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.342208 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b04709\"/\"ipp-route-b04709-kserve-self-signed-certs\"" Jul 24 16:53:31.347978 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.347948 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd"] Jul 24 16:53:31.367562 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.367539 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:53:31.474745 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.474716 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-dshm\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.474912 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.474770 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711137a9-36c8-43a9-881e-84d72183e8fb-tls-certs\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.474912 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.474803 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8c5\" (UniqueName: \"kubernetes.io/projected/711137a9-36c8-43a9-881e-84d72183e8fb-kube-api-access-fr8c5\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.474912 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.474865 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-tmp-dir\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.475039 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.474912 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-home\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.475039 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.474932 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-model-cache\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.575673 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.575644 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-home\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.575849 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.575680 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-model-cache\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.575849 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.575751 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-dshm\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.575849 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.575796 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711137a9-36c8-43a9-881e-84d72183e8fb-tls-certs\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.575849 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.575819 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8c5\" (UniqueName: \"kubernetes.io/projected/711137a9-36c8-43a9-881e-84d72183e8fb-kube-api-access-fr8c5\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.576077 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.575853 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-tmp-dir\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.576137 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.576101 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-home\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.576137 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.576130 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-model-cache\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.576290 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.576250 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-tmp-dir\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.578018 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.577999 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-dshm\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.578333 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.578317 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711137a9-36c8-43a9-881e-84d72183e8fb-tls-certs\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.584074 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.584051 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8c5\" (UniqueName: \"kubernetes.io/projected/711137a9-36c8-43a9-881e-84d72183e8fb-kube-api-access-fr8c5\") pod \"ipp-route-b04709-kserve-7ccd9944cf-klcpd\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.650889 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.650858 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:31.785583 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:31.785563 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd"] Jul 24 16:53:31.788805 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:53:31.788778 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod711137a9_36c8_43a9_881e_84d72183e8fb.slice/crio-f479e586c62a74ac18b85c63ce23f440d0b3146c861cdaab47503c5fcd0539aa WatchSource:0}: Error finding container f479e586c62a74ac18b85c63ce23f440d0b3146c861cdaab47503c5fcd0539aa: Status 404 returned error can't find the container with id f479e586c62a74ac18b85c63ce23f440d0b3146c861cdaab47503c5fcd0539aa Jul 24 16:53:32.274279 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:32.274240 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" event={"ID":"711137a9-36c8-43a9-881e-84d72183e8fb","Type":"ContainerStarted","Data":"08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1"} Jul 24 16:53:32.274279 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:32.274278 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" event={"ID":"711137a9-36c8-43a9-881e-84d72183e8fb","Type":"ContainerStarted","Data":"f479e586c62a74ac18b85c63ce23f440d0b3146c861cdaab47503c5fcd0539aa"} Jul 24 16:53:32.294532 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:32.294487 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" podStartSLOduration=1.2944706049999999 podStartE2EDuration="1.294470605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:53:32.292050064 +0000 UTC m=+2831.723685431" watchObservedRunningTime="2026-07-24 16:53:32.294470605 +0000 UTC m=+2831.726105965" Jul 24 16:53:41.651217 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:41.651179 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:41.651632 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:41.651234 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:41.663650 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:41.663625 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:53:42.340463 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:53:42.340385 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:54:02.774927 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:02.774895 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9l5g2_6a9cb2fb-a288-44d2-b490-f90c66e90120/payload-processing/0.log" Jul 24 16:54:03.152515 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:03.152485 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:03.460048 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:03.459969 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:03.960531 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:03.960502 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:06.748395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:06.748361 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:06.844377 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:06.844320 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:08.225050 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.225017 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d"] Jul 24 16:54:08.228964 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.228946 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.231330 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.231312 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8c969d-serving-cert\"" Jul 24 16:54:08.240053 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.240012 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d"] Jul 24 16:54:08.265359 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.265330 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:08.328384 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.328359 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55c3d88-ed1c-4aeb-8230-3f50393203e8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.328540 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.328390 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55c3d88-ed1c-4aeb-8230-3f50393203e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.328540 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.328501 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvhg8\" (UniqueName: \"kubernetes.io/projected/f55c3d88-ed1c-4aeb-8230-3f50393203e8-kube-api-access-tvhg8\") pod \"maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.363739 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.363708 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:08.429586 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.429561 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvhg8\" (UniqueName: \"kubernetes.io/projected/f55c3d88-ed1c-4aeb-8230-3f50393203e8-kube-api-access-tvhg8\") pod \"maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.429708 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.429597 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55c3d88-ed1c-4aeb-8230-3f50393203e8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.429708 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.429619 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55c3d88-ed1c-4aeb-8230-3f50393203e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.430349 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.430333 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55c3d88-ed1c-4aeb-8230-3f50393203e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.432196 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.432178 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55c3d88-ed1c-4aeb-8230-3f50393203e8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.438448 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.438429 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvhg8\" (UniqueName: \"kubernetes.io/projected/f55c3d88-ed1c-4aeb-8230-3f50393203e8-kube-api-access-tvhg8\") pod \"maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.541028 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.540953 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:08.716278 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.716230 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d"] Jul 24 16:54:08.718729 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:54:08.718699 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55c3d88_ed1c_4aeb_8230_3f50393203e8.slice/crio-73b539308cb1f469490596eeff6d2f85f44b3dcac9d1def644938aff53fc2b06 WatchSource:0}: Error finding container 73b539308cb1f469490596eeff6d2f85f44b3dcac9d1def644938aff53fc2b06: Status 404 returned error can't find the container with id 73b539308cb1f469490596eeff6d2f85f44b3dcac9d1def644938aff53fc2b06 Jul 24 16:54:08.720118 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.720097 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:54:08.769355 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.769329 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:08.877430 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:08.877399 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:09.464121 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:09.464024 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" event={"ID":"f55c3d88-ed1c-4aeb-8230-3f50393203e8","Type":"ContainerStarted","Data":"a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78"} Jul 24 16:54:09.464121 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:09.464117 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" event={"ID":"f55c3d88-ed1c-4aeb-8230-3f50393203e8","Type":"ContainerStarted","Data":"73b539308cb1f469490596eeff6d2f85f44b3dcac9d1def644938aff53fc2b06"} Jul 24 16:54:09.464538 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:09.464167 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:09.481093 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:09.481043 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" podStartSLOduration=1.011359254 podStartE2EDuration="1.481027667s" totalImagesPullingTime="469.668413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:54:09.479063914 +0000 UTC m=+2868.910699275" watchObservedRunningTime="2026-07-24 16:54:09.481027667 +0000 UTC m=+2868.912663026" Jul 24 16:54:11.143995 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:11.143923 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:15.474038 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:15.474009 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:22.052988 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.052947 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt"] Jul 24 16:54:22.057512 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.057485 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt"] Jul 24 16:54:22.057687 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.057611 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" Jul 24 16:54:22.062018 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.061995 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtjsm\"" Jul 24 16:54:22.166837 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.166799 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb850f-87e0-4c76-a86a-8cda18196749-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt\" (UID: \"4dfb850f-87e0-4c76-a86a-8cda18196749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" Jul 24 16:54:22.268476 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.268438 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb850f-87e0-4c76-a86a-8cda18196749-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt\" (UID: \"4dfb850f-87e0-4c76-a86a-8cda18196749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" Jul 24 16:54:22.268964 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.268944 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb850f-87e0-4c76-a86a-8cda18196749-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt\" (UID: \"4dfb850f-87e0-4c76-a86a-8cda18196749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" Jul 24 16:54:22.369392 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.369365 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" Jul 24 16:54:22.518238 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.518175 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt"] Jul 24 16:54:22.520689 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:54:22.520661 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfb850f_87e0_4c76_a86a_8cda18196749.slice/crio-f1cd12d7156b9ffdc10853991b0756b38e0b4773fcd3480ded24f602390d2a83 WatchSource:0}: Error finding container f1cd12d7156b9ffdc10853991b0756b38e0b4773fcd3480ded24f602390d2a83: Status 404 returned error can't find the container with id f1cd12d7156b9ffdc10853991b0756b38e0b4773fcd3480ded24f602390d2a83 Jul 24 16:54:22.530543 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:22.530518 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" event={"ID":"4dfb850f-87e0-4c76-a86a-8cda18196749","Type":"ContainerStarted","Data":"f1cd12d7156b9ffdc10853991b0756b38e0b4773fcd3480ded24f602390d2a83"} Jul 24 16:54:23.537162 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:23.537052 2560 generic.go:356] "Generic (PLEG): container finished" podID="4dfb850f-87e0-4c76-a86a-8cda18196749" containerID="65e8989b2dd8e088fc0b18be975ef5c038ceb9aae59594ba4fdf7aa5e289003a" exitCode=0 Jul 24 16:54:23.537516 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:23.537160 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" event={"ID":"4dfb850f-87e0-4c76-a86a-8cda18196749","Type":"ContainerDied","Data":"65e8989b2dd8e088fc0b18be975ef5c038ceb9aae59594ba4fdf7aa5e289003a"} Jul 24 16:54:24.676083 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:24.676061 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" Jul 24 16:54:24.794469 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:24.794387 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb850f-87e0-4c76-a86a-8cda18196749-maas-api-service-ca\") pod \"4dfb850f-87e0-4c76-a86a-8cda18196749\" (UID: \"4dfb850f-87e0-4c76-a86a-8cda18196749\") " Jul 24 16:54:24.794739 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:24.794719 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfb850f-87e0-4c76-a86a-8cda18196749-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4dfb850f-87e0-4c76-a86a-8cda18196749" (UID: "4dfb850f-87e0-4c76-a86a-8cda18196749"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:54:24.895834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:24.895805 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb850f-87e0-4c76-a86a-8cda18196749-maas-api-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:54:25.549220 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:25.549110 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" Jul 24 16:54:25.549220 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:25.549165 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt" event={"ID":"4dfb850f-87e0-4c76-a86a-8cda18196749","Type":"ContainerDied","Data":"f1cd12d7156b9ffdc10853991b0756b38e0b4773fcd3480ded24f602390d2a83"} Jul 24 16:54:25.549220 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:25.549218 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1cd12d7156b9ffdc10853991b0756b38e0b4773fcd3480ded24f602390d2a83" Jul 24 16:54:32.954898 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:32.954861 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d"] Jul 24 16:54:32.955444 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:32.955180 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" podUID="f55c3d88-ed1c-4aeb-8230-3f50393203e8" containerName="maas-api" containerID="cri-o://a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78" gracePeriod=30 Jul 24 16:54:32.992671 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:32.992640 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:33.091948 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.091913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:33.212608 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.212550 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:33.379585 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.379555 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvhg8\" (UniqueName: \"kubernetes.io/projected/f55c3d88-ed1c-4aeb-8230-3f50393203e8-kube-api-access-tvhg8\") pod \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " Jul 24 16:54:33.379783 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.379732 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55c3d88-ed1c-4aeb-8230-3f50393203e8-openshift-service-ca\") pod \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " Jul 24 16:54:33.379783 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.379770 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55c3d88-ed1c-4aeb-8230-3f50393203e8-maas-api-tls\") pod \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\" (UID: \"f55c3d88-ed1c-4aeb-8230-3f50393203e8\") " Jul 24 16:54:33.380101 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.380075 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55c3d88-ed1c-4aeb-8230-3f50393203e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f55c3d88-ed1c-4aeb-8230-3f50393203e8" (UID: "f55c3d88-ed1c-4aeb-8230-3f50393203e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:54:33.382055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.382022 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55c3d88-ed1c-4aeb-8230-3f50393203e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f55c3d88-ed1c-4aeb-8230-3f50393203e8" (UID: "f55c3d88-ed1c-4aeb-8230-3f50393203e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:54:33.382168 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.382080 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55c3d88-ed1c-4aeb-8230-3f50393203e8-kube-api-access-tvhg8" (OuterVolumeSpecName: "kube-api-access-tvhg8") pod "f55c3d88-ed1c-4aeb-8230-3f50393203e8" (UID: "f55c3d88-ed1c-4aeb-8230-3f50393203e8"). InnerVolumeSpecName "kube-api-access-tvhg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:54:33.481410 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.481327 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55c3d88-ed1c-4aeb-8230-3f50393203e8-openshift-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:54:33.481410 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.481354 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55c3d88-ed1c-4aeb-8230-3f50393203e8-maas-api-tls\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:54:33.481410 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.481364 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvhg8\" (UniqueName: \"kubernetes.io/projected/f55c3d88-ed1c-4aeb-8230-3f50393203e8-kube-api-access-tvhg8\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:54:33.593433 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.593399 2560 generic.go:356] "Generic (PLEG): container finished" podID="f55c3d88-ed1c-4aeb-8230-3f50393203e8" containerID="a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78" exitCode=0 Jul 24 16:54:33.593720 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.593459 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" Jul 24 16:54:33.593720 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.593483 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" event={"ID":"f55c3d88-ed1c-4aeb-8230-3f50393203e8","Type":"ContainerDied","Data":"a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78"} Jul 24 16:54:33.593720 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.593544 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d" event={"ID":"f55c3d88-ed1c-4aeb-8230-3f50393203e8","Type":"ContainerDied","Data":"73b539308cb1f469490596eeff6d2f85f44b3dcac9d1def644938aff53fc2b06"} Jul 24 16:54:33.593720 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.593575 2560 scope.go:117] "RemoveContainer" containerID="a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78" Jul 24 16:54:33.603931 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.603915 2560 scope.go:117] "RemoveContainer" containerID="a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78" Jul 24 16:54:33.604192 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:54:33.604174 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78\": container with ID starting with a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78 not found: ID does not exist" containerID="a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78" Jul 24 16:54:33.604260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.604201 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78"} err="failed to get container status \"a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78\": rpc error: code = NotFound desc = could not find container \"a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78\": container with ID starting with a2a4b53286230feac4a2c9339cad6116bc38d5898c1b3e5a01f2471b36b3ba78 not found: ID does not exist" Jul 24 16:54:33.615083 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.615062 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d"] Jul 24 16:54:33.618276 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:33.618258 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8c969d-58cdc84f74-t2g6d"] Jul 24 16:54:35.166940 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:35.166908 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55c3d88-ed1c-4aeb-8230-3f50393203e8" path="/var/lib/kubelet/pods/f55c3d88-ed1c-4aeb-8230-3f50393203e8/volumes" Jul 24 16:54:37.116549 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:37.116520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:37.129112 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:37.129078 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt"] Jul 24 16:54:37.130939 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:37.130921 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8c969d-pz2rt"] Jul 24 16:54:37.167191 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:37.167133 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dfb850f-87e0-4c76-a86a-8cda18196749" path="/var/lib/kubelet/pods/4dfb850f-87e0-4c76-a86a-8cda18196749/volumes" Jul 24 16:54:37.506742 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:37.506657 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:43.528452 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:43.528420 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:43.629774 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:43.629746 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:44.134447 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:44.134415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:44.657368 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:44.657340 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:54:50.395373 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.395331 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch"] Jul 24 16:54:50.395789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.395759 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55c3d88-ed1c-4aeb-8230-3f50393203e8" containerName="maas-api" Jul 24 16:54:50.395789 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.395773 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55c3d88-ed1c-4aeb-8230-3f50393203e8" containerName="maas-api" Jul 24 16:54:50.395866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.395828 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dfb850f-87e0-4c76-a86a-8cda18196749" containerName="revoke-keys" Jul 24 16:54:50.395866 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.395839 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dfb850f-87e0-4c76-a86a-8cda18196749" containerName="revoke-keys" Jul 24 16:54:50.395929 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.395920 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dfb850f-87e0-4c76-a86a-8cda18196749" containerName="revoke-keys" Jul 24 16:54:50.395963 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.395933 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55c3d88-ed1c-4aeb-8230-3f50393203e8" containerName="maas-api" Jul 24 16:54:50.400897 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.400879 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" Jul 24 16:54:50.403110 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.403093 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtjsm\"" Jul 24 16:54:50.408071 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.408046 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch"] Jul 24 16:54:50.438000 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.437970 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc48fca2-5464-45d8-b25f-957719564e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch\" (UID: \"cc48fca2-5464-45d8-b25f-957719564e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" Jul 24 16:54:50.538508 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.538473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc48fca2-5464-45d8-b25f-957719564e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch\" (UID: \"cc48fca2-5464-45d8-b25f-957719564e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" Jul 24 16:54:50.539052 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.539035 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc48fca2-5464-45d8-b25f-957719564e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch\" (UID: \"cc48fca2-5464-45d8-b25f-957719564e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" Jul 24 16:54:50.712836 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:50.712762 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" Jul 24 16:54:51.054788 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:51.054758 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch"] Jul 24 16:54:51.057458 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:54:51.057430 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc48fca2_5464_45d8_b25f_957719564e6d.slice/crio-e2057bece12eff358691322295ead13a8de32e75c7134622046ef5427fbae29a WatchSource:0}: Error finding container e2057bece12eff358691322295ead13a8de32e75c7134622046ef5427fbae29a: Status 404 returned error can't find the container with id e2057bece12eff358691322295ead13a8de32e75c7134622046ef5427fbae29a Jul 24 16:54:51.686996 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:51.686901 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" event={"ID":"cc48fca2-5464-45d8-b25f-957719564e6d","Type":"ContainerStarted","Data":"e4f0739c023717559800be36455e31e1a31659e67dcd961dada507e5f703d280"} Jul 24 16:54:51.686996 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:51.686972 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" event={"ID":"cc48fca2-5464-45d8-b25f-957719564e6d","Type":"ContainerStarted","Data":"e2057bece12eff358691322295ead13a8de32e75c7134622046ef5427fbae29a"} Jul 24 16:54:51.706796 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:51.706741 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" podStartSLOduration=1.7067263000000001 podStartE2EDuration="1.7067263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:54:51.704379222 +0000 UTC m=+2911.136014583" watchObservedRunningTime="2026-07-24 16:54:51.7067263 +0000 UTC m=+2911.138361661" Jul 24 16:54:52.694293 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:52.694257 2560 generic.go:356] "Generic (PLEG): container finished" podID="cc48fca2-5464-45d8-b25f-957719564e6d" containerID="e4f0739c023717559800be36455e31e1a31659e67dcd961dada507e5f703d280" exitCode=0 Jul 24 16:54:52.694660 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:52.694302 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" event={"ID":"cc48fca2-5464-45d8-b25f-957719564e6d","Type":"ContainerDied","Data":"e4f0739c023717559800be36455e31e1a31659e67dcd961dada507e5f703d280"} Jul 24 16:54:53.834157 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:53.834116 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" Jul 24 16:54:53.867726 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:53.867697 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc48fca2-5464-45d8-b25f-957719564e6d-maas-api-service-ca\") pod \"cc48fca2-5464-45d8-b25f-957719564e6d\" (UID: \"cc48fca2-5464-45d8-b25f-957719564e6d\") " Jul 24 16:54:53.868012 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:53.867993 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc48fca2-5464-45d8-b25f-957719564e6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc48fca2-5464-45d8-b25f-957719564e6d" (UID: "cc48fca2-5464-45d8-b25f-957719564e6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:54:53.969023 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:53.968953 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc48fca2-5464-45d8-b25f-957719564e6d-maas-api-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:54:54.706730 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:54.706694 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" Jul 24 16:54:54.706730 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:54.706714 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch" event={"ID":"cc48fca2-5464-45d8-b25f-957719564e6d","Type":"ContainerDied","Data":"e2057bece12eff358691322295ead13a8de32e75c7134622046ef5427fbae29a"} Jul 24 16:54:54.706941 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:54:54.706758 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2057bece12eff358691322295ead13a8de32e75c7134622046ef5427fbae29a" Jul 24 16:55:05.691260 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:05.691226 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:05.791452 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:05.791421 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:05.844520 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:05.844489 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:10.467464 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:10.467425 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch"] Jul 24 16:55:10.472622 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:10.472592 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b04709-r2cch"] Jul 24 16:55:10.473621 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:10.473601 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:10.864942 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:10.864904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:11.168956 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:11.168879 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc48fca2-5464-45d8-b25f-957719564e6d" path="/var/lib/kubelet/pods/cc48fca2-5464-45d8-b25f-957719564e6d/volumes" Jul 24 16:55:15.857984 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:15.857953 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd"] Jul 24 16:55:15.954337 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:15.954297 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:30.877903 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:30.877840 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" podUID="711137a9-36c8-43a9-881e-84d72183e8fb" containerName="main" containerID="cri-o://08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1" gracePeriod=45 Jul 24 16:55:31.130465 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.130410 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:55:31.232416 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232387 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr8c5\" (UniqueName: \"kubernetes.io/projected/711137a9-36c8-43a9-881e-84d72183e8fb-kube-api-access-fr8c5\") pod \"711137a9-36c8-43a9-881e-84d72183e8fb\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " Jul 24 16:55:31.232416 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232421 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-model-cache\") pod \"711137a9-36c8-43a9-881e-84d72183e8fb\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " Jul 24 16:55:31.232650 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232473 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711137a9-36c8-43a9-881e-84d72183e8fb-tls-certs\") pod \"711137a9-36c8-43a9-881e-84d72183e8fb\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " Jul 24 16:55:31.232650 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232499 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-dshm\") pod \"711137a9-36c8-43a9-881e-84d72183e8fb\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " Jul 24 16:55:31.232650 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232529 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-tmp-dir\") pod \"711137a9-36c8-43a9-881e-84d72183e8fb\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " Jul 24 16:55:31.232650 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232551 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-home\") pod \"711137a9-36c8-43a9-881e-84d72183e8fb\" (UID: \"711137a9-36c8-43a9-881e-84d72183e8fb\") " Jul 24 16:55:31.232862 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232741 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "711137a9-36c8-43a9-881e-84d72183e8fb" (UID: "711137a9-36c8-43a9-881e-84d72183e8fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:55:31.232862 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232839 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "711137a9-36c8-43a9-881e-84d72183e8fb" (UID: "711137a9-36c8-43a9-881e-84d72183e8fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:55:31.232958 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232884 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-home" (OuterVolumeSpecName: "home") pod "711137a9-36c8-43a9-881e-84d72183e8fb" (UID: "711137a9-36c8-43a9-881e-84d72183e8fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:55:31.232958 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.232917 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-model-cache\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:55:31.234824 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.234792 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-dshm" (OuterVolumeSpecName: "dshm") pod "711137a9-36c8-43a9-881e-84d72183e8fb" (UID: "711137a9-36c8-43a9-881e-84d72183e8fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:55:31.234935 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.234840 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711137a9-36c8-43a9-881e-84d72183e8fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "711137a9-36c8-43a9-881e-84d72183e8fb" (UID: "711137a9-36c8-43a9-881e-84d72183e8fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:55:31.234935 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.234852 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711137a9-36c8-43a9-881e-84d72183e8fb-kube-api-access-fr8c5" (OuterVolumeSpecName: "kube-api-access-fr8c5") pod "711137a9-36c8-43a9-881e-84d72183e8fb" (UID: "711137a9-36c8-43a9-881e-84d72183e8fb"). InnerVolumeSpecName "kube-api-access-fr8c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:55:31.333606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.333569 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8c5\" (UniqueName: \"kubernetes.io/projected/711137a9-36c8-43a9-881e-84d72183e8fb-kube-api-access-fr8c5\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:55:31.333606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.333600 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711137a9-36c8-43a9-881e-84d72183e8fb-tls-certs\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:55:31.333606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.333610 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-dshm\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:55:31.333606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.333617 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-tmp-dir\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:55:31.333872 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.333624 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711137a9-36c8-43a9-881e-84d72183e8fb-home\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:55:31.900305 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.900272 2560 generic.go:356] "Generic (PLEG): container finished" podID="711137a9-36c8-43a9-881e-84d72183e8fb" containerID="08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1" exitCode=0 Jul 24 16:55:31.900719 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.900327 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" Jul 24 16:55:31.900719 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.900346 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" event={"ID":"711137a9-36c8-43a9-881e-84d72183e8fb","Type":"ContainerDied","Data":"08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1"} Jul 24 16:55:31.900719 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.900382 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd" event={"ID":"711137a9-36c8-43a9-881e-84d72183e8fb","Type":"ContainerDied","Data":"f479e586c62a74ac18b85c63ce23f440d0b3146c861cdaab47503c5fcd0539aa"} Jul 24 16:55:31.900719 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.900402 2560 scope.go:117] "RemoveContainer" containerID="08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1" Jul 24 16:55:31.909442 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.909426 2560 scope.go:117] "RemoveContainer" containerID="08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1" Jul 24 16:55:31.909686 ip-10-0-134-103 kubenswrapper[2560]: E0724 16:55:31.909668 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1\": container with ID starting with 08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1 not found: ID does not exist" containerID="08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1" Jul 24 16:55:31.909742 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.909694 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1"} err="failed to get container status \"08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1\": rpc error: code = NotFound desc = could not find container \"08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1\": container with ID starting with 08f68852ee4e1c38ab5d2b2fc49455160e42b7c4b3b3a174eb392219023180d1 not found: ID does not exist" Jul 24 16:55:31.921917 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.921896 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd"] Jul 24 16:55:31.925763 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:31.925744 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b04709/ipp-route-b04709-kserve-7ccd9944cf-klcpd"] Jul 24 16:55:33.166718 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:33.166682 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711137a9-36c8-43a9-881e-84d72183e8fb" path="/var/lib/kubelet/pods/711137a9-36c8-43a9-881e-84d72183e8fb/volumes" Jul 24 16:55:37.724657 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:37.724620 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:37.825833 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:37.825805 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:52.739722 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:52.739692 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:52.870865 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:52.870831 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:58.507762 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:58.507730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:58.565214 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:58.565180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:58.782649 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:58.782566 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:59.020835 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.020801 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:55:59.143208 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.143178 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c"] Jul 24 16:55:59.143737 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.143718 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc48fca2-5464-45d8-b25f-957719564e6d" containerName="revoke-keys" Jul 24 16:55:59.143825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.143740 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc48fca2-5464-45d8-b25f-957719564e6d" containerName="revoke-keys" Jul 24 16:55:59.143825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.143757 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711137a9-36c8-43a9-881e-84d72183e8fb" containerName="main" Jul 24 16:55:59.143825 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.143765 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="711137a9-36c8-43a9-881e-84d72183e8fb" containerName="main" Jul 24 16:55:59.143941 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.143848 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc48fca2-5464-45d8-b25f-957719564e6d" containerName="revoke-keys" Jul 24 16:55:59.143941 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.143865 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="711137a9-36c8-43a9-881e-84d72183e8fb" containerName="main" Jul 24 16:55:59.146921 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.146888 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" Jul 24 16:55:59.149603 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.149533 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtjsm\"" Jul 24 16:55:59.151213 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.151191 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c"] Jul 24 16:55:59.293339 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.293307 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9fe3d-dc9e-48fb-978e-7679fc253118-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c\" (UID: \"19d9fe3d-dc9e-48fb-978e-7679fc253118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" Jul 24 16:55:59.394605 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.394515 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9fe3d-dc9e-48fb-978e-7679fc253118-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c\" (UID: \"19d9fe3d-dc9e-48fb-978e-7679fc253118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" Jul 24 16:55:59.395104 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.395084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9fe3d-dc9e-48fb-978e-7679fc253118-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c\" (UID: \"19d9fe3d-dc9e-48fb-978e-7679fc253118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" Jul 24 16:55:59.459121 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.459096 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" Jul 24 16:55:59.597965 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:55:59.597938 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c"] Jul 24 16:55:59.601349 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:55:59.601319 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d9fe3d_dc9e_48fb_978e_7679fc253118.slice/crio-dcffa5db531dbe2d0ea375446aa7c84ce99d620636b66478c732289fba0c5378 WatchSource:0}: Error finding container dcffa5db531dbe2d0ea375446aa7c84ce99d620636b66478c732289fba0c5378: Status 404 returned error can't find the container with id dcffa5db531dbe2d0ea375446aa7c84ce99d620636b66478c732289fba0c5378 Jul 24 16:56:00.049653 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:00.049560 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" event={"ID":"19d9fe3d-dc9e-48fb-978e-7679fc253118","Type":"ContainerStarted","Data":"620c6161d6dfc2e08cb49242ac90e5475901bbb134cb143337c677ae002ff4b8"} Jul 24 16:56:00.049653 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:00.049625 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" event={"ID":"19d9fe3d-dc9e-48fb-978e-7679fc253118","Type":"ContainerStarted","Data":"dcffa5db531dbe2d0ea375446aa7c84ce99d620636b66478c732289fba0c5378"} Jul 24 16:56:00.064940 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:00.064883 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" podStartSLOduration=1.064868237 podStartE2EDuration="1.064868237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:56:00.063241959 +0000 UTC m=+2979.494877320" watchObservedRunningTime="2026-07-24 16:56:00.064868237 +0000 UTC m=+2979.496503596" Jul 24 16:56:01.056205 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:01.056171 2560 generic.go:356] "Generic (PLEG): container finished" podID="19d9fe3d-dc9e-48fb-978e-7679fc253118" containerID="620c6161d6dfc2e08cb49242ac90e5475901bbb134cb143337c677ae002ff4b8" exitCode=0 Jul 24 16:56:01.056599 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:01.056257 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" event={"ID":"19d9fe3d-dc9e-48fb-978e-7679fc253118","Type":"ContainerDied","Data":"620c6161d6dfc2e08cb49242ac90e5475901bbb134cb143337c677ae002ff4b8"} Jul 24 16:56:02.193178 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:02.193129 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" Jul 24 16:56:02.226373 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:02.226346 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9fe3d-dc9e-48fb-978e-7679fc253118-maas-api-service-ca\") pod \"19d9fe3d-dc9e-48fb-978e-7679fc253118\" (UID: \"19d9fe3d-dc9e-48fb-978e-7679fc253118\") " Jul 24 16:56:02.226661 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:02.226639 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d9fe3d-dc9e-48fb-978e-7679fc253118-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19d9fe3d-dc9e-48fb-978e-7679fc253118" (UID: "19d9fe3d-dc9e-48fb-978e-7679fc253118"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:56:02.327174 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:02.327080 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d9fe3d-dc9e-48fb-978e-7679fc253118-maas-api-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:56:03.067275 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:03.067242 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" Jul 24 16:56:03.067456 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:03.067278 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c" event={"ID":"19d9fe3d-dc9e-48fb-978e-7679fc253118","Type":"ContainerDied","Data":"dcffa5db531dbe2d0ea375446aa7c84ce99d620636b66478c732289fba0c5378"} Jul 24 16:56:03.067456 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:03.067352 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcffa5db531dbe2d0ea375446aa7c84ce99d620636b66478c732289fba0c5378" Jul 24 16:56:09.584559 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:09.584520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:09.684531 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:09.684495 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:09.800237 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:09.800208 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:13.803895 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:13.803864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:13.855966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:13.855933 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:14.211773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:14.211736 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c"] Jul 24 16:56:14.213839 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:14.213807 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:14.215253 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:14.215231 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-633b54-btr9c"] Jul 24 16:56:14.625466 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:14.625427 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:15.166055 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:15.166025 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d9fe3d-dc9e-48fb-978e-7679fc253118" path="/var/lib/kubelet/pods/19d9fe3d-dc9e-48fb-978e-7679fc253118/volumes" Jul 24 16:56:20.418130 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:20.418097 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:20.479118 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:20.479090 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:20.699297 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:20.699205 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:20.952775 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:20.952692 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:21.106660 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.106626 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj"] Jul 24 16:56:21.107046 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.107034 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d9fe3d-dc9e-48fb-978e-7679fc253118" containerName="revoke-keys" Jul 24 16:56:21.107091 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.107047 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d9fe3d-dc9e-48fb-978e-7679fc253118" containerName="revoke-keys" Jul 24 16:56:21.107126 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.107115 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d9fe3d-dc9e-48fb-978e-7679fc253118" containerName="revoke-keys" Jul 24 16:56:21.110585 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.110569 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" Jul 24 16:56:21.113078 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.113061 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtjsm\"" Jul 24 16:56:21.117015 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.116993 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj"] Jul 24 16:56:21.206121 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.206041 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4fcd5a-b408-423f-83e1-2536c68bd43c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj\" (UID: \"af4fcd5a-b408-423f-83e1-2536c68bd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" Jul 24 16:56:21.235190 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.235163 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:56:21.246046 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.246019 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlf24_b9ef7edf-3ea8-4270-b7c7-acf95e550718/ovn-acl-logging/0.log" Jul 24 16:56:21.306944 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.306911 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4fcd5a-b408-423f-83e1-2536c68bd43c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj\" (UID: \"af4fcd5a-b408-423f-83e1-2536c68bd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" Jul 24 16:56:21.307492 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.307467 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4fcd5a-b408-423f-83e1-2536c68bd43c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj\" (UID: \"af4fcd5a-b408-423f-83e1-2536c68bd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" Jul 24 16:56:21.422694 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.422668 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" Jul 24 16:56:21.767478 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:21.767454 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj"] Jul 24 16:56:21.769789 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:56:21.769761 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4fcd5a_b408_423f_83e1_2536c68bd43c.slice/crio-b9474bb0df501b21d8c3dda865a63cfd1da81696fcbe59398822b7a5d19c1074 WatchSource:0}: Error finding container b9474bb0df501b21d8c3dda865a63cfd1da81696fcbe59398822b7a5d19c1074: Status 404 returned error can't find the container with id b9474bb0df501b21d8c3dda865a63cfd1da81696fcbe59398822b7a5d19c1074 Jul 24 16:56:22.167921 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:22.167885 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" event={"ID":"af4fcd5a-b408-423f-83e1-2536c68bd43c","Type":"ContainerStarted","Data":"dbd68e430469fda783b55bd6b60e4c6721320caa29320e5f317ed1e7c4e4b68c"} Jul 24 16:56:22.168081 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:22.167944 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" event={"ID":"af4fcd5a-b408-423f-83e1-2536c68bd43c","Type":"ContainerStarted","Data":"b9474bb0df501b21d8c3dda865a63cfd1da81696fcbe59398822b7a5d19c1074"} Jul 24 16:56:22.182250 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:22.182196 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" podStartSLOduration=1.182176815 podStartE2EDuration="1.182176815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:56:22.181584081 +0000 UTC m=+3001.613219433" watchObservedRunningTime="2026-07-24 16:56:22.182176815 +0000 UTC m=+3001.613812177" Jul 24 16:56:23.173809 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:23.173777 2560 generic.go:356] "Generic (PLEG): container finished" podID="af4fcd5a-b408-423f-83e1-2536c68bd43c" containerID="dbd68e430469fda783b55bd6b60e4c6721320caa29320e5f317ed1e7c4e4b68c" exitCode=0 Jul 24 16:56:23.174218 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:23.173868 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" event={"ID":"af4fcd5a-b408-423f-83e1-2536c68bd43c","Type":"ContainerDied","Data":"dbd68e430469fda783b55bd6b60e4c6721320caa29320e5f317ed1e7c4e4b68c"} Jul 24 16:56:24.310777 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:24.310752 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" Jul 24 16:56:24.441249 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:24.441175 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4fcd5a-b408-423f-83e1-2536c68bd43c-maas-api-service-ca\") pod \"af4fcd5a-b408-423f-83e1-2536c68bd43c\" (UID: \"af4fcd5a-b408-423f-83e1-2536c68bd43c\") " Jul 24 16:56:24.441481 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:24.441462 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4fcd5a-b408-423f-83e1-2536c68bd43c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af4fcd5a-b408-423f-83e1-2536c68bd43c" (UID: "af4fcd5a-b408-423f-83e1-2536c68bd43c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:56:24.542195 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:24.542165 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4fcd5a-b408-423f-83e1-2536c68bd43c-maas-api-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:56:25.184526 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:25.184495 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" Jul 24 16:56:25.184670 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:25.184500 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj" event={"ID":"af4fcd5a-b408-423f-83e1-2536c68bd43c","Type":"ContainerDied","Data":"b9474bb0df501b21d8c3dda865a63cfd1da81696fcbe59398822b7a5d19c1074"} Jul 24 16:56:25.184670 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:25.184617 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9474bb0df501b21d8c3dda865a63cfd1da81696fcbe59398822b7a5d19c1074" Jul 24 16:56:31.775438 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:31.775401 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:31.847100 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:31.847067 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:31.914905 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:31.914869 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:36.178915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:36.178865 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:36.186677 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:36.186647 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj"] Jul 24 16:56:36.190594 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:36.190570 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a65dee-xrlsj"] Jul 24 16:56:36.560436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:36.560352 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:37.167118 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:37.167078 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4fcd5a-b408-423f-83e1-2536c68bd43c" path="/var/lib/kubelet/pods/af4fcd5a-b408-423f-83e1-2536c68bd43c/volumes" Jul 24 16:56:42.587381 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:42.587344 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:42.786871 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:42.786836 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:42.905882 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:42.905828 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:43.274990 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.274904 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx"] Jul 24 16:56:43.275531 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.275511 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4fcd5a-b408-423f-83e1-2536c68bd43c" containerName="revoke-keys" Jul 24 16:56:43.275631 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.275533 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4fcd5a-b408-423f-83e1-2536c68bd43c" containerName="revoke-keys" Jul 24 16:56:43.275686 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.275639 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4fcd5a-b408-423f-83e1-2536c68bd43c" containerName="revoke-keys" Jul 24 16:56:43.278899 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.278877 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" Jul 24 16:56:43.281299 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.281283 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtjsm\"" Jul 24 16:56:43.286326 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.286294 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx"] Jul 24 16:56:43.296959 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.296927 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:43.317817 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.317792 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249b1c78-8a7f-4b30-8fe5-5540c8eb19b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx\" (UID: \"249b1c78-8a7f-4b30-8fe5-5540c8eb19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" Jul 24 16:56:43.419102 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.419073 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249b1c78-8a7f-4b30-8fe5-5540c8eb19b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx\" (UID: \"249b1c78-8a7f-4b30-8fe5-5540c8eb19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" Jul 24 16:56:43.419725 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.419706 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249b1c78-8a7f-4b30-8fe5-5540c8eb19b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx\" (UID: \"249b1c78-8a7f-4b30-8fe5-5540c8eb19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" Jul 24 16:56:43.592088 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.592015 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" Jul 24 16:56:43.731526 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:43.731499 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx"] Jul 24 16:56:43.733682 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:56:43.733641 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249b1c78_8a7f_4b30_8fe5_5540c8eb19b6.slice/crio-167d00660a8ed96f73d09cb473b503e87568f4bc096de29fcf8684f962b892fb WatchSource:0}: Error finding container 167d00660a8ed96f73d09cb473b503e87568f4bc096de29fcf8684f962b892fb: Status 404 returned error can't find the container with id 167d00660a8ed96f73d09cb473b503e87568f4bc096de29fcf8684f962b892fb Jul 24 16:56:44.281962 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:44.281867 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" event={"ID":"249b1c78-8a7f-4b30-8fe5-5540c8eb19b6","Type":"ContainerStarted","Data":"1133a31a4fb2bf8177201cb2ece8b155be1abb0b60baefee5240b34066387f0a"} Jul 24 16:56:44.281962 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:44.281939 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" event={"ID":"249b1c78-8a7f-4b30-8fe5-5540c8eb19b6","Type":"ContainerStarted","Data":"167d00660a8ed96f73d09cb473b503e87568f4bc096de29fcf8684f962b892fb"} Jul 24 16:56:44.298206 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:44.298139 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" podStartSLOduration=1.298125995 podStartE2EDuration="1.298125995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:56:44.295728029 +0000 UTC m=+3023.727363391" watchObservedRunningTime="2026-07-24 16:56:44.298125995 +0000 UTC m=+3023.729761354" Jul 24 16:56:45.288810 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:45.288771 2560 generic.go:356] "Generic (PLEG): container finished" podID="249b1c78-8a7f-4b30-8fe5-5540c8eb19b6" containerID="1133a31a4fb2bf8177201cb2ece8b155be1abb0b60baefee5240b34066387f0a" exitCode=0 Jul 24 16:56:45.289207 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:45.288853 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" event={"ID":"249b1c78-8a7f-4b30-8fe5-5540c8eb19b6","Type":"ContainerDied","Data":"1133a31a4fb2bf8177201cb2ece8b155be1abb0b60baefee5240b34066387f0a"} Jul 24 16:56:46.430584 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:46.430559 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" Jul 24 16:56:46.447189 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:46.447139 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249b1c78-8a7f-4b30-8fe5-5540c8eb19b6-maas-api-service-ca\") pod \"249b1c78-8a7f-4b30-8fe5-5540c8eb19b6\" (UID: \"249b1c78-8a7f-4b30-8fe5-5540c8eb19b6\") " Jul 24 16:56:46.447880 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:46.447838 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249b1c78-8a7f-4b30-8fe5-5540c8eb19b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "249b1c78-8a7f-4b30-8fe5-5540c8eb19b6" (UID: "249b1c78-8a7f-4b30-8fe5-5540c8eb19b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:56:46.548788 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:46.548709 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/249b1c78-8a7f-4b30-8fe5-5540c8eb19b6-maas-api-service-ca\") on node \"ip-10-0-134-103.ec2.internal\" DevicePath \"\"" Jul 24 16:56:46.757436 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:46.757402 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:47.300516 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:47.300475 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" Jul 24 16:56:47.300516 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:47.300511 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx" event={"ID":"249b1c78-8a7f-4b30-8fe5-5540c8eb19b6","Type":"ContainerDied","Data":"167d00660a8ed96f73d09cb473b503e87568f4bc096de29fcf8684f962b892fb"} Jul 24 16:56:47.300783 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:47.300572 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="167d00660a8ed96f73d09cb473b503e87568f4bc096de29fcf8684f962b892fb" Jul 24 16:56:53.642304 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:53.642265 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:53.741831 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:53.741798 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:58.341036 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:58.341008 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:58.345807 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:58.345781 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx"] Jul 24 16:56:58.348281 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:58.348258 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a29da6-sjkvx"] Jul 24 16:56:58.727379 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:58.727346 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:56:59.168013 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:56:59.167981 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="249b1c78-8a7f-4b30-8fe5-5540c8eb19b6" path="/var/lib/kubelet/pods/249b1c78-8a7f-4b30-8fe5-5540c8eb19b6/volumes" Jul 24 16:57:04.541389 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:04.541352 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:57:04.597484 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:04.597450 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:57:04.822980 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:04.822887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:57:05.054456 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:05.054421 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:57:05.474578 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:05.474549 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b78dd6555-4wjhj_dd62a736-38ba-4386-9af1-1497e68ae17d/authorino/0.log" Jul 24 16:57:11.061308 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:11.061274 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8c774b47-9mrg4_b561f2fa-cb00-4c0e-8e69-eac1419f35a1/manager/0.log" Jul 24 16:57:11.407302 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:11.407276 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d6c795b6-5t7s9_bdd2b47f-636d-4e3f-b397-92a44e4268b7/copy-manifests/0.log" Jul 24 16:57:11.414445 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:11.414423 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d6c795b6-5t7s9_bdd2b47f-636d-4e3f-b397-92a44e4268b7/manager/0.log" Jul 24 16:57:12.029969 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:12.029932 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d79565ccd-qpcvl_b4ed83a5-f630-48eb-bea4-fd4c5f5763c9/maas-api/0.log" Jul 24 16:57:12.357834 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:12.357816 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hn6cf_31911853-3917-4c5f-a1e1-8b911098de4c/postgres/0.log" Jul 24 16:57:13.634024 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:13.633990 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b78dd6555-4wjhj_dd62a736-38ba-4386-9af1-1497e68ae17d/authorino/0.log" Jul 24 16:57:13.982759 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:13.982687 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-w59lp_38632bce-1983-4157-87d4-e2c99662e222/kuadrant-console-plugin/0.log" Jul 24 16:57:15.154476 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:15.154439 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd59bb69-wqxgh_6fc0c646-2f90-4fc0-b497-6322d0f84a12/kube-auth-proxy/0.log" Jul 24 16:57:15.523295 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:15.523216 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9l5g2_6a9cb2fb-a288-44d2-b490-f90c66e90120/payload-processing/0.log" Jul 24 16:57:15.991860 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:15.991827 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bcdsl_4cea08c2-8201-459d-b5be-70636e547578/main/0.log" Jul 24 16:57:16.336979 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:16.336901 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvwpdm_6d8d03ae-516b-4d87-8429-e1f111d41915/main/0.log" Jul 24 16:57:22.203522 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:22.203484 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:57:22.266981 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:22.266943 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:57:25.192588 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:25.192559 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7gtd8_6ea154fa-ed3f-4f3c-b3d4-980666719a5d/global-pull-secret-syncer/0.log" Jul 24 16:57:25.285048 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:25.285020 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" Jul 24 16:57:25.417929 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:25.417898 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-103.ec2.internal_46b3df3b80b90b643851b1c22d7ab10a/haproxy/0.log" Jul 24 16:57:30.198072 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:30.198033 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b78dd6555-4wjhj_dd62a736-38ba-4386-9af1-1497e68ae17d/authorino/0.log" Jul 24 16:57:30.275849 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:30.275819 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-w59lp_38632bce-1983-4157-87d4-e2c99662e222/kuadrant-console-plugin/0.log" Jul 24 16:57:32.507643 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.507613 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c527801d-3221-40bc-8acf-42cd957d008c/alertmanager/0.log" Jul 24 16:57:32.530179 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.530130 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c527801d-3221-40bc-8acf-42cd957d008c/config-reloader/0.log" Jul 24 16:57:32.558694 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.558670 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c527801d-3221-40bc-8acf-42cd957d008c/kube-rbac-proxy-web/0.log" Jul 24 16:57:32.584414 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.584389 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c527801d-3221-40bc-8acf-42cd957d008c/kube-rbac-proxy/0.log" Jul 24 16:57:32.605242 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.605219 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c527801d-3221-40bc-8acf-42cd957d008c/kube-rbac-proxy-metric/0.log" Jul 24 16:57:32.625202 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.625177 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c527801d-3221-40bc-8acf-42cd957d008c/prom-label-proxy/0.log" Jul 24 16:57:32.648737 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.648719 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c527801d-3221-40bc-8acf-42cd957d008c/init-config-reloader/0.log" Jul 24 16:57:32.739430 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.739348 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-5nw7f_3421be4c-1658-4f37-b7b4-363b802d0c69/kube-state-metrics/0.log" Jul 24 16:57:32.758444 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.758419 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-5nw7f_3421be4c-1658-4f37-b7b4-363b802d0c69/kube-rbac-proxy-main/0.log" Jul 24 16:57:32.779628 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.779602 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-5nw7f_3421be4c-1658-4f37-b7b4-363b802d0c69/kube-rbac-proxy-self/0.log" Jul 24 16:57:32.813228 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.813203 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f9844cddc-ksj4r_d75a1be4-84f6-44b9-95a0-55e2d7d5ea3e/metrics-server/0.log" Jul 24 16:57:32.839608 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.839591 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-7tnn9_a8b4e365-3ff3-45f8-8914-9236e5aa033d/monitoring-plugin/0.log" Jul 24 16:57:32.957548 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.957521 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-szpx2_da368237-4ba5-4ddd-905c-c61f0ca37d08/node-exporter/0.log" Jul 24 16:57:32.984537 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:32.984515 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-szpx2_da368237-4ba5-4ddd-905c-c61f0ca37d08/kube-rbac-proxy/0.log" Jul 24 16:57:33.007610 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.007542 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-szpx2_da368237-4ba5-4ddd-905c-c61f0ca37d08/init-textfile/0.log" Jul 24 16:57:33.112793 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.112768 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-9qvsm_abbc9b98-206f-41d7-b91a-cc701b707d4e/kube-rbac-proxy-main/0.log" Jul 24 16:57:33.133637 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.133609 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-9qvsm_abbc9b98-206f-41d7-b91a-cc701b707d4e/kube-rbac-proxy-self/0.log" Jul 24 16:57:33.157768 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.157740 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-9qvsm_abbc9b98-206f-41d7-b91a-cc701b707d4e/openshift-state-metrics/0.log" Jul 24 16:57:33.196600 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.196576 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17b7e2ee-207b-4fdc-9e59-b112fd10843a/prometheus/0.log" Jul 24 16:57:33.216777 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.216754 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17b7e2ee-207b-4fdc-9e59-b112fd10843a/config-reloader/0.log" Jul 24 16:57:33.238420 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.238398 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17b7e2ee-207b-4fdc-9e59-b112fd10843a/thanos-sidecar/0.log" Jul 24 16:57:33.259687 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.259629 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17b7e2ee-207b-4fdc-9e59-b112fd10843a/kube-rbac-proxy-web/0.log" Jul 24 16:57:33.281199 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.281179 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17b7e2ee-207b-4fdc-9e59-b112fd10843a/kube-rbac-proxy/0.log" Jul 24 16:57:33.302664 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.302643 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17b7e2ee-207b-4fdc-9e59-b112fd10843a/kube-rbac-proxy-thanos/0.log" Jul 24 16:57:33.324371 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.324351 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17b7e2ee-207b-4fdc-9e59-b112fd10843a/init-config-reloader/0.log" Jul 24 16:57:33.352327 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.352304 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-r2qp6_9bd7f79f-23d4-41f9-afbc-c6ee87318297/prometheus-operator/0.log" Jul 24 16:57:33.372603 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.372583 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-r2qp6_9bd7f79f-23d4-41f9-afbc-c6ee87318297/kube-rbac-proxy/0.log" Jul 24 16:57:33.394670 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.394647 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-zlgc6_eab9cdfe-4b71-428f-b037-7b85300f7f30/prometheus-operator-admission-webhook/0.log" Jul 24 16:57:33.510915 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.510844 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769c878fbf-9zdq6_f319b5fc-461d-4665-a37a-5317b77e9bf9/thanos-query/0.log" Jul 24 16:57:33.530814 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.530788 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769c878fbf-9zdq6_f319b5fc-461d-4665-a37a-5317b77e9bf9/kube-rbac-proxy-web/0.log" Jul 24 16:57:33.556728 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.556696 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769c878fbf-9zdq6_f319b5fc-461d-4665-a37a-5317b77e9bf9/kube-rbac-proxy/0.log" Jul 24 16:57:33.581074 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.581049 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769c878fbf-9zdq6_f319b5fc-461d-4665-a37a-5317b77e9bf9/prom-label-proxy/0.log" Jul 24 16:57:33.602395 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.602375 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769c878fbf-9zdq6_f319b5fc-461d-4665-a37a-5317b77e9bf9/kube-rbac-proxy-rules/0.log" Jul 24 16:57:33.621770 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:33.621741 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-769c878fbf-9zdq6_f319b5fc-461d-4665-a37a-5317b77e9bf9/kube-rbac-proxy-metrics/0.log" Jul 24 16:57:35.405262 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.405223 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w"] Jul 24 16:57:35.405654 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.405641 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249b1c78-8a7f-4b30-8fe5-5540c8eb19b6" containerName="revoke-keys" Jul 24 16:57:35.405699 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.405655 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="249b1c78-8a7f-4b30-8fe5-5540c8eb19b6" containerName="revoke-keys" Jul 24 16:57:35.405739 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.405730 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="249b1c78-8a7f-4b30-8fe5-5540c8eb19b6" containerName="revoke-keys" Jul 24 16:57:35.408995 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.408973 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.411629 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.411608 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vptbp\"/\"openshift-service-ca.crt\"" Jul 24 16:57:35.411763 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.411643 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vptbp\"/\"kube-root-ca.crt\"" Jul 24 16:57:35.412608 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.412592 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vptbp\"/\"default-dockercfg-r859z\"" Jul 24 16:57:35.415934 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.415915 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w"] Jul 24 16:57:35.507769 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.507738 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-podres\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.507966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.507776 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwd9\" (UniqueName: \"kubernetes.io/projected/21659d68-1ea3-4972-b3a0-2df93e23e7f6-kube-api-access-6pwd9\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.507966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.507853 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-proc\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.507966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.507889 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-sys\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.507966 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.507963 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-lib-modules\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.609428 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.609393 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-lib-modules\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.609606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.609446 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-podres\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.609606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.609465 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwd9\" (UniqueName: \"kubernetes.io/projected/21659d68-1ea3-4972-b3a0-2df93e23e7f6-kube-api-access-6pwd9\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.609606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.609504 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-proc\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.609606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.609530 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-sys\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.609606 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.609584 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-lib-modules\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.609810 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.609584 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-podres\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.609810 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.609622 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-proc\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.609810 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.609632 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21659d68-1ea3-4972-b3a0-2df93e23e7f6-sys\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.617002 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.616980 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwd9\" (UniqueName: \"kubernetes.io/projected/21659d68-1ea3-4972-b3a0-2df93e23e7f6-kube-api-access-6pwd9\") pod \"perf-node-gather-daemonset-r5w5w\" (UID: \"21659d68-1ea3-4972-b3a0-2df93e23e7f6\") " pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.720571 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.720495 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:35.865480 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:35.865449 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w"] Jul 24 16:57:35.868848 ip-10-0-134-103 kubenswrapper[2560]: W0724 16:57:35.868814 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21659d68_1ea3_4972_b3a0_2df93e23e7f6.slice/crio-3e0c1bb2e1f53526a4cc8856d5ae4fdf5f17b4baba6a0fce9b7ccb14f0649c4b WatchSource:0}: Error finding container 3e0c1bb2e1f53526a4cc8856d5ae4fdf5f17b4baba6a0fce9b7ccb14f0649c4b: Status 404 returned error can't find the container with id 3e0c1bb2e1f53526a4cc8856d5ae4fdf5f17b4baba6a0fce9b7ccb14f0649c4b Jul 24 16:57:36.005960 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:36.005874 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cdc4c448b-6srp6_71ef4113-c810-4121-bbdc-22dffe8a3fc2/console/0.log" Jul 24 16:57:36.038107 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:36.038080 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-xmnf6_e514e66f-f0cb-4119-b3a2-555d1d6702be/download-server/0.log" Jul 24 16:57:36.557425 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:36.557389 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" event={"ID":"21659d68-1ea3-4972-b3a0-2df93e23e7f6","Type":"ContainerStarted","Data":"d0dbb77b5b89fe02ee6e7a838bb1cabed019f6a21f883c275b0dc8d43fa52626"} Jul 24 16:57:36.557888 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:36.557445 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" event={"ID":"21659d68-1ea3-4972-b3a0-2df93e23e7f6","Type":"ContainerStarted","Data":"3e0c1bb2e1f53526a4cc8856d5ae4fdf5f17b4baba6a0fce9b7ccb14f0649c4b"} Jul 24 16:57:36.557888 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:36.557506 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:36.574575 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:36.574531 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" podStartSLOduration=1.574519239 podStartE2EDuration="1.574519239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:57:36.571937632 +0000 UTC m=+3076.003573185" watchObservedRunningTime="2026-07-24 16:57:36.574519239 +0000 UTC m=+3076.006154599" Jul 24 16:57:37.717983 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:37.717951 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pmbd4_e6ad1888-b16b-4aee-9925-95a654c4ff65/dns/0.log" Jul 24 16:57:37.742715 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:37.742688 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pmbd4_e6ad1888-b16b-4aee-9925-95a654c4ff65/kube-rbac-proxy/0.log" Jul 24 16:57:37.820251 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:37.820226 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xwds8_65297386-a516-4da5-b689-c3e03c004310/dns-node-resolver/0.log" Jul 24 16:57:38.382173 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:38.382131 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlv94_95229c4e-2eb9-4401-8282-0cfd32c54496/node-ca/0.log" Jul 24 16:57:39.514041 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:39.514010 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd59bb69-wqxgh_6fc0c646-2f90-4fc0-b497-6322d0f84a12/kube-auth-proxy/0.log" Jul 24 16:57:39.675625 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:39.675584 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9l5g2_6a9cb2fb-a288-44d2-b490-f90c66e90120/payload-processing/0.log" Jul 24 16:57:40.516121 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:40.516078 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qr5k7_6a059404-9f78-4a76-a3d0-38bdb8969166/serve-healthcheck-canary/0.log" Jul 24 16:57:41.155566 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:41.155518 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x2mw4_a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d/kube-rbac-proxy/0.log" Jul 24 16:57:41.176042 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:41.176013 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x2mw4_a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d/exporter/0.log" Jul 24 16:57:41.196496 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:41.196455 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x2mw4_a5022dcf-82e1-4dca-bcd0-e1fdd8aacf6d/extractor/0.log" Jul 24 16:57:41.917115 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:41.917080 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87tpp_6f746255-6327-410b-82e0-2a3b60e801fb/konnectivity-agent/0.log" limit=1024 Jul 24 16:57:42.572659 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:42.572628 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vptbp/perf-node-gather-daemonset-r5w5w" Jul 24 16:57:43.815038 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:43.815000 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8c774b47-9mrg4_b561f2fa-cb00-4c0e-8e69-eac1419f35a1/manager/0.log" Jul 24 16:57:44.002553 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:44.002503 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d6c795b6-5t7s9_bdd2b47f-636d-4e3f-b397-92a44e4268b7/manager/0.log" Jul 24 16:57:44.025971 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:44.025946 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d6c795b6-5t7s9_bdd2b47f-636d-4e3f-b397-92a44e4268b7/copy-manifests/0.log" Jul 24 16:57:45.616417 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:45.616387 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9cffb5c7d-45vrj_b104b9a1-fb54-4f03-a35e-53b6d37a8ff6/manager/0.log" Jul 24 16:57:45.666938 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:45.666910 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hppwl_c636941c-b9c2-4f0a-98ba-e0bdec7e723a/openshift-lws-operator/0.log" Jul 24 16:57:45.859305 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:45.859281 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd59bb69-wqxgh_6fc0c646-2f90-4fc0-b497-6322d0f84a12/kube-auth-proxy/0.log" Jul 24 16:57:45.985790 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:45.985704 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9l5g2_6a9cb2fb-a288-44d2-b490-f90c66e90120/payload-processing/0.log" Jul 24 16:57:47.607482 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:47.607459 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd59bb69-wqxgh_6fc0c646-2f90-4fc0-b497-6322d0f84a12/kube-auth-proxy/0.log" Jul 24 16:57:47.742468 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:47.742437 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9l5g2_6a9cb2fb-a288-44d2-b490-f90c66e90120/payload-processing/0.log" Jul 24 16:57:52.207250 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.207217 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kf2n5_c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5/kube-multus-additional-cni-plugins/0.log" Jul 24 16:57:52.227912 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.227887 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kf2n5_c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5/egress-router-binary-copy/0.log" Jul 24 16:57:52.250336 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.250301 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kf2n5_c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5/cni-plugins/0.log" Jul 24 16:57:52.270466 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.270442 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kf2n5_c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5/bond-cni-plugin/0.log" Jul 24 16:57:52.290638 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.290612 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kf2n5_c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5/routeoverride-cni/0.log" Jul 24 16:57:52.310773 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.310739 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kf2n5_c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5/whereabouts-cni-bincopy/0.log" Jul 24 16:57:52.331479 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.331455 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kf2n5_c3dd5400-8bd9-4cda-ab6d-fbaed2420bf5/whereabouts-cni/0.log" Jul 24 16:57:52.568707 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.568627 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mlv5n_4ffb9abd-d6d8-43d2-a3ef-23bd8d1ac46d/kube-multus/0.log" Jul 24 16:57:52.631131 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.631103 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9lbnz_cd77de22-56d1-4095-8674-7613919037b0/network-metrics-daemon/0.log" Jul 24 16:57:52.649457 ip-10-0-134-103 kubenswrapper[2560]: I0724 16:57:52.649418 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9lbnz_cd77de22-56d1-4095-8674-7613919037b0/kube-rbac-proxy/0.log"