Jul 06 12:23:02.691445 ip-10-0-136-228 systemd[1]: Starting Kubernetes Kubelet... Jul 06 12:23:05.614193 ip-10-0-136-228 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 12:23:05.614193 ip-10-0-136-228 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 12:23:05.614193 ip-10-0-136-228 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 12:23:05.614193 ip-10-0-136-228 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 12:23:05.614193 ip-10-0-136-228 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 12:23:05.614831 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.614207 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 12:23:05.617759 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617744 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:23:05.617759 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617758 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617762 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617765 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617768 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617771 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617774 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617776 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617779 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617782 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617784 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617787 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617790 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617792 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617795 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617798 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617800 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617803 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617806 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617808 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617811 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:23:05.617818 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617814 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617816 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617818 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617821 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617824 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617827 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617829 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617832 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617835 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617837 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617840 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617842 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617845 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617847 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617850 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617853 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617855 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617857 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617861 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617863 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:23:05.618263 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617866 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617868 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617871 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617873 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617876 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617878 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617881 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617883 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617886 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617888 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617891 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617894 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617896 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617899 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617902 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617904 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617907 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617909 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617912 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617914 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:23:05.618755 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617917 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617920 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617922 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617925 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617927 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617930 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617932 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617935 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617939 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617943 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617946 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617949 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617951 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617954 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617956 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617959 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617961 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617964 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617968 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:23:05.619209 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617972 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617975 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617977 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617981 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617984 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617987 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617989 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617993 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617996 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.617999 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618001 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618004 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618006 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618009 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618013 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618016 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618018 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618021 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618024 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:23:05.619649 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618732 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618740 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618745 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618749 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618753 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618756 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618759 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618762 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618765 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618768 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618771 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618773 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618776 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618778 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618781 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618784 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618786 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618789 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618792 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:23:05.620103 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618794 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618797 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618800 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618802 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618805 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618808 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618810 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618813 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618816 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618818 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618820 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618823 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618826 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618828 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618831 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618833 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618836 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618839 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618841 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:23:05.620553 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618844 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618846 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618848 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618851 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618853 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618856 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618858 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618861 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618864 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618867 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618870 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618872 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618875 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618877 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618881 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618884 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618886 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618888 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618891 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618893 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:23:05.621014 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618896 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618898 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618901 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618903 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618905 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618908 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618911 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618913 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618916 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618918 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618920 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618923 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618926 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618928 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618931 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618933 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618936 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618938 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618941 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618943 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:23:05.621480 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618947 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618956 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618959 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618963 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618966 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618969 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618972 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618975 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618977 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618980 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618983 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618985 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618987 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618990 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618992 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618995 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.618997 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619000 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619002 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:23:05.621959 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619004 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619007 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619402 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619409 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619412 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619415 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619418 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619421 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619424 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619426 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619429 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619432 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619436 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619439 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619443 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619447 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619450 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619452 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619456 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619459 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:23:05.622404 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619462 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619464 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619467 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619469 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619472 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619475 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619477 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619480 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619482 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619484 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619487 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619489 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619491 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619494 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619496 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619499 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619501 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619504 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619507 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619510 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:23:05.622918 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619512 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619516 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619519 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619523 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619526 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619529 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619534 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619537 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619540 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619543 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619545 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619548 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619551 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619554 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619556 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619559 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619561 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619563 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619566 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:23:05.623393 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619569 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619571 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619574 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619576 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619579 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619581 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619583 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619594 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619597 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619600 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619603 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619605 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619608 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619610 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619612 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619615 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619618 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619620 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619623 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619626 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:23:05.623882 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619632 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619634 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619637 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619639 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619642 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619645 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619648 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619651 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619653 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619670 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619674 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619677 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619680 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619682 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619685 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619687 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619690 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619693 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619695 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619698 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:23:05.624392 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619700 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.619703 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619766 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619772 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619777 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619782 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619786 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619789 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619793 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619797 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619800 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619806 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619810 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619813 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619816 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619820 2572 flags.go:64] FLAG: --cgroup-root="" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619822 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619825 2572 flags.go:64] FLAG: --client-ca-file="" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619829 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619832 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619835 2572 flags.go:64] FLAG: --cluster-domain="" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619838 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619841 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619844 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 12:23:05.624886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619848 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619851 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619854 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619857 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619860 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619863 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619866 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619869 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619872 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619875 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619879 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619882 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619885 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619888 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619891 2572 flags.go:64] FLAG: --enable-server="true" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619894 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619897 2572 flags.go:64] FLAG: --event-burst="100" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619900 2572 flags.go:64] FLAG: --event-qps="50" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619903 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619907 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619910 2572 flags.go:64] FLAG: --eviction-hard="" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619914 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619917 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619919 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619922 2572 flags.go:64] FLAG: --eviction-soft="" Jul 06 12:23:05.625425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619925 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619929 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619932 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619935 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619938 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619941 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619944 2572 flags.go:64] FLAG: --feature-gates="" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619947 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619950 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619953 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619956 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619959 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619962 2572 flags.go:64] FLAG: --help="false" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619965 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-136-228.ec2.internal" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619968 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619970 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619973 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619977 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619980 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619982 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619985 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619988 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619992 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 12:23:05.626091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619994 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.619997 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620001 2572 flags.go:64] FLAG: --kube-reserved="" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620005 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620008 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620011 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620014 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620017 2572 flags.go:64] FLAG: --lock-file="" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620020 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620023 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620026 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620031 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620034 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620037 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620040 2572 flags.go:64] FLAG: --logging-format="text" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620042 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620046 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620048 2572 flags.go:64] FLAG: --manifest-url="" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620051 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620055 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620058 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620062 2572 flags.go:64] FLAG: --max-pods="110" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620065 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620068 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620071 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 12:23:05.626688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620074 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620077 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620080 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620083 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620090 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620093 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620095 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620098 2572 flags.go:64] FLAG: --pod-cidr="" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620101 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620111 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620115 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620118 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620121 2572 flags.go:64] FLAG: --port="10250" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620124 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620127 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c9071f57ff76799f" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620130 2572 flags.go:64] FLAG: --qos-reserved="" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620133 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620136 2572 flags.go:64] FLAG: --register-node="true" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620138 2572 flags.go:64] FLAG: --register-with-taints="" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620142 2572 flags.go:64] FLAG: --registry-burst="10" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620145 2572 flags.go:64] FLAG: --registry-qps="5" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620148 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620151 2572 flags.go:64] FLAG: --reserved-memory="" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620155 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620157 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 12:23:05.627297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620161 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620164 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620166 2572 flags.go:64] FLAG: --runonce="false" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620169 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620172 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620175 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620178 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620180 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620184 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620186 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620190 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620193 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620195 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620198 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620201 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620204 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620207 2572 flags.go:64] FLAG: --system-cgroups="" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620209 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620215 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620219 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620222 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620226 2572 flags.go:64] FLAG: --tls-min-version="" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620228 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620231 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620234 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 12:23:05.627933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620237 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620239 2572 flags.go:64] FLAG: --v="2" Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620246 2572 flags.go:64] FLAG: --version="false" Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620251 2572 flags.go:64] FLAG: --vmodule="" Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620255 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.620258 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620372 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620375 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620378 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620381 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620383 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620386 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620388 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620391 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620393 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620395 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620398 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620400 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620403 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620405 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620408 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:23:05.628904 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620410 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620413 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620416 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620418 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620421 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620425 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620427 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620430 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620432 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620435 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620437 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620440 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620442 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620445 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620447 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620451 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620453 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620455 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620458 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620461 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620463 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:23:05.629476 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620466 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620468 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620471 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620473 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620476 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620478 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620481 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620485 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620488 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620491 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620493 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620496 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620498 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620501 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620504 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620506 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620510 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620513 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620515 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:23:05.630048 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620518 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620520 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620523 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620525 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620528 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620530 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620534 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620538 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620542 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620544 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620548 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620551 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620553 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620556 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620559 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620562 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620564 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620567 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620569 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:23:05.630495 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620572 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620575 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620577 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620579 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620582 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620584 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620587 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620589 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620592 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620594 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620598 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620601 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620603 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620606 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620609 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620611 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620614 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620616 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620619 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620621 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:23:05.630965 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620624 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620628 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620631 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620635 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.620638 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.621251 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.629855 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.629870 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.629889 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.629896 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629951 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629956 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629959 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629962 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:23:05.631443 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629965 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629967 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629970 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629973 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629975 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629978 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629981 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629984 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629987 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629989 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629992 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629994 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.629997 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630006 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630009 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630011 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630014 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630016 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630019 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:23:05.631812 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630021 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630024 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630027 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630029 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630032 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630034 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630037 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630041 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630045 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630049 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630052 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630055 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630058 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630061 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630064 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630067 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630070 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630072 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630075 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:23:05.632256 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630078 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630080 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630083 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630086 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630089 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630093 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630096 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630100 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630102 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630105 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630107 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630110 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630113 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630115 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630118 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630120 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630123 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630125 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630128 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630131 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:23:05.632736 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630134 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630136 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630139 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630142 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630144 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630147 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630150 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630152 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630155 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630157 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630160 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630162 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630164 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630167 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630169 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630172 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630174 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630177 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630180 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630183 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:23:05.633198 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630185 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630188 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630190 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630193 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630195 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630198 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630202 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630205 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630208 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630211 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630214 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630218 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630221 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630224 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630227 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630230 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:23:05.633677 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630233 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.630237 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630348 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630353 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630355 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630358 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630361 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630363 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630366 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630369 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630372 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630374 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630377 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630379 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630382 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:23:05.634057 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630385 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630388 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630391 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630393 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630396 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630399 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630401 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630404 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630406 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630409 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630411 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630414 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630418 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630421 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630423 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630426 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630429 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630431 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630433 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:23:05.634419 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630436 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630438 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630441 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630443 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630446 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630448 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630451 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630454 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630456 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630459 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630461 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630464 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630466 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630481 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630484 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630486 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630489 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630491 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630494 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630496 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:23:05.634926 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630499 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630502 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630505 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630509 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630512 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630516 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630519 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630521 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630524 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630526 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630529 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630532 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630534 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630537 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630539 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630542 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630544 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630547 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630550 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630553 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:23:05.635407 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630555 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630558 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630561 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630565 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630568 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630572 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630576 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630579 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630582 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630585 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630587 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630590 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630592 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630595 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630597 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630600 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630602 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630605 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630607 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:23:05.636010 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630610 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630612 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630615 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630617 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630620 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630622 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630625 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:05.630627 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.630633 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.630748 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.632769 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.634149 2572 server.go:1019] "Starting client certificate rotation" Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.634247 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 12:23:05.636454 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.634287 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 12:23:05.636788 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.636637 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 12:23:05.637826 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.637806 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 12:23:05.651636 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.651613 2572 log.go:25] "Validated CRI v1 runtime API" Jul 06 12:23:05.653989 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.653976 2572 log.go:25] "Validated CRI v1 image API" Jul 06 12:23:05.654781 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.654760 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 12:23:05.658088 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.658067 2572 fs.go:135] Filesystem UUIDs: map[00501553-6978-4a0d-83bc-512b454136d3:/dev/nvme0n1p3 586ec430-5d76-4765-8783-13a00b991169:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 06 12:23:05.658168 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.658088 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 12:23:05.662835 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.662725 2572 manager.go:216] Machine: {Timestamp:2026-07-06 12:23:05.662236571 +0000 UTC m=+1.740784589 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104190 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29b79aa853431ff17383e1775b07ec SystemUUID:ec29b79a-a853-431f-f173-83e1775b07ec BootID:c8445b41-f068-4dba-a3b5-8159e91a059c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:76:d3:c4:b4:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:76:d3:c4:b4:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:e9:72:f2:38:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 12:23:05.662835 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.662826 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 12:23:05.662955 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.662933 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 12:23:05.663687 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.663649 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 12:23:05.665688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.665628 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 12:23:05.665831 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.665669 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 12:23:05.666498 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.666486 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 12:23:05.666529 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.666500 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 12:23:05.666529 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.666507 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 12:23:05.667188 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.667178 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 12:23:05.668467 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.668457 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 12:23:05.671295 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.671285 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 06 12:23:05.671407 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.671398 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 12:23:05.671438 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.671428 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 06 12:23:05.671466 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.671441 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 12:23:05.671466 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.671455 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 12:23:05.671466 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.671464 2572 kubelet.go:398] "Adding apiserver pod source" Jul 06 12:23:05.671574 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.671474 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 12:23:05.672432 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.672411 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 12:23:05.672523 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.672442 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 12:23:05.674985 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.674966 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 12:23:05.676987 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.676974 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 12:23:05.677047 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.677004 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 12:23:05.678158 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678147 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 12:23:05.678191 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678166 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 12:23:05.678191 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678174 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 12:23:05.678191 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678179 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 12:23:05.678191 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678188 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 12:23:05.678191 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678193 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 12:23:05.678321 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678202 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 12:23:05.678321 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678207 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 12:23:05.678321 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678214 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 12:23:05.678321 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678219 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 12:23:05.678321 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678226 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 12:23:05.678321 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.678231 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 12:23:05.679601 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.679583 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 12:23:05.679601 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.679601 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 12:23:05.682621 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.682609 2572 server.go:1268] "Started kubelet" Jul 06 12:23:05.682781 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.682702 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 12:23:05.682873 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.682783 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 12:23:05.683509 ip-10-0-136-228 systemd[1]: Started Kubernetes Kubelet. Jul 06 12:23:05.684288 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.684227 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 12:23:05.684485 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.684462 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 12:23:05.684485 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.684469 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 12:23:05.684594 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.684520 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 12:23:05.684791 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.684591 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 12:23:05.685970 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.685957 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 06 12:23:05.688058 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.687084 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-228.ec2.internal.18bfb2d06b25b947 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-228.ec2.internal,UID:ip-10-0-136-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-228.ec2.internal,},FirstTimestamp:2026-07-06 12:23:05.682581831 +0000 UTC m=+1.761129849,LastTimestamp:2026-07-06 12:23:05.682581831 +0000 UTC m=+1.761129849,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-228.ec2.internal,}" Jul 06 12:23:05.689274 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.689256 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 12:23:05.689894 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.689874 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 12:23:05.689982 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.689892 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 12:23:05.690036 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.689976 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:05.690533 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.690511 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 12:23:05.690618 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.690571 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 12:23:05.690618 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.690589 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 12:23:05.691073 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691052 2572 factory.go:55] Registering systemd factory Jul 06 12:23:05.691073 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691078 2572 factory.go:223] Registration of the systemd container factory successfully Jul 06 12:23:05.691210 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691199 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 06 12:23:05.691210 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691209 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 06 12:23:05.691311 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691296 2572 factory.go:153] Registering CRI-O factory Jul 06 12:23:05.691311 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691307 2572 factory.go:223] Registration of the crio container factory successfully Jul 06 12:23:05.691374 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691358 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 12:23:05.691406 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691386 2572 factory.go:103] Registering Raw factory Jul 06 12:23:05.691406 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691401 2572 manager.go:1196] Started watching for new ooms in manager Jul 06 12:23:05.691804 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.691793 2572 manager.go:349] Starting recovery of all containers Jul 06 12:23:05.695900 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.695869 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 12:23:05.696016 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.695993 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 12:23:05.702392 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.702374 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-np74v" Jul 06 12:23:05.703083 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.703067 2572 manager.go:354] Recovery completed Jul 06 12:23:05.707437 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.707426 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:23:05.709284 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.709270 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:23:05.709329 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.709294 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:23:05.709329 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.709306 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:23:05.709804 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.709791 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 12:23:05.709804 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.709804 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 12:23:05.709926 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.709822 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 06 12:23:05.709926 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.709831 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:23:05.710510 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.710498 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:23:05.710575 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.710519 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:23:05.710575 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.710534 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:23:05.711053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.711040 2572 policy_none.go:49] "None policy: Start" Jul 06 12:23:05.711104 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.711058 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 12:23:05.711104 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.711071 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 12:23:05.711203 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.711189 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-np74v" Jul 06 12:23:05.712444 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.712385 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-228.ec2.internal.18bfb2d06cbd2813 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-228.ec2.internal,UID:ip-10-0-136-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-228.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-228.ec2.internal,},FirstTimestamp:2026-07-06 12:23:05.709283347 +0000 UTC m=+1.787831365,LastTimestamp:2026-07-06 12:23:05.709283347 +0000 UTC m=+1.787831365,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-228.ec2.internal,}" Jul 06 12:23:05.712614 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.712602 2572 policy_none.go:47] "Start" Jul 06 12:23:05.744216 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.744199 2572 manager.go:337] "Starting Device Plugin manager" Jul 06 12:23:05.744287 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.744231 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 12:23:05.744287 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.744240 2572 server.go:87] "Starting device plugin registration server" Jul 06 12:23:05.744453 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.744443 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 12:23:05.744494 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.744454 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 12:23:05.744563 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.744545 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 12:23:05.744637 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.744626 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 12:23:05.744698 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.744637 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 12:23:05.745773 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.745753 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 12:23:05.745844 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.745799 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:05.823887 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.823846 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 12:23:05.825067 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.825050 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 12:23:05.825146 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.825089 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 12:23:05.825146 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.825110 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 12:23:05.825146 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.825143 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 12:23:05.829156 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.829137 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:23:05.845458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.845442 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:23:05.846210 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.846196 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:23:05.846286 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.846227 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:23:05.846286 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.846240 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:23:05.846286 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.846264 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-228.ec2.internal" Jul 06 12:23:05.856888 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.856875 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-228.ec2.internal" Jul 06 12:23:05.856932 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.856897 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-228.ec2.internal\": node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:05.878810 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.878766 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:05.926133 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.926107 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"] Jul 06 12:23:05.926253 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.926165 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:23:05.927008 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.926994 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:23:05.927078 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.927021 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:23:05.927078 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.927037 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:23:05.928315 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.928302 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:23:05.928462 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.928444 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" Jul 06 12:23:05.928524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.928481 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:23:05.929053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.929037 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:23:05.929053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.929047 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:23:05.929182 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.929066 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:23:05.929182 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.929077 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:23:05.929182 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.929082 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:23:05.929182 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.929095 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:23:05.930302 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.930289 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" Jul 06 12:23:05.930367 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.930313 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:23:05.931031 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.931016 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:23:05.931097 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.931051 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:23:05.931097 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.931061 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:23:05.956285 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.956267 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-228.ec2.internal\" not found" node="ip-10-0-136-228.ec2.internal" Jul 06 12:23:05.960582 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.960568 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-228.ec2.internal\" not found" node="ip-10-0-136-228.ec2.internal" Jul 06 12:23:05.979419 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:05.979402 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:05.992772 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.992754 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bb0dda8697e373c519f6dc1fe8e110b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-228.ec2.internal\" (UID: \"9bb0dda8697e373c519f6dc1fe8e110b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" Jul 06 12:23:05.992842 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.992778 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1de034bb34c18d0d39fa5bf9bef22dd8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"1de034bb34c18d0d39fa5bf9bef22dd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" Jul 06 12:23:05.992842 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:05.992797 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1de034bb34c18d0d39fa5bf9bef22dd8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"1de034bb34c18d0d39fa5bf9bef22dd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.080385 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.080344 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:06.093860 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.093833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bb0dda8697e373c519f6dc1fe8e110b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-228.ec2.internal\" (UID: \"9bb0dda8697e373c519f6dc1fe8e110b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.093961 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.093883 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1de034bb34c18d0d39fa5bf9bef22dd8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"1de034bb34c18d0d39fa5bf9bef22dd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.093961 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.093907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1de034bb34c18d0d39fa5bf9bef22dd8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"1de034bb34c18d0d39fa5bf9bef22dd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.093961 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.093835 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bb0dda8697e373c519f6dc1fe8e110b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-228.ec2.internal\" (UID: \"9bb0dda8697e373c519f6dc1fe8e110b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.093961 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.093951 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1de034bb34c18d0d39fa5bf9bef22dd8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"1de034bb34c18d0d39fa5bf9bef22dd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.094097 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.093965 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1de034bb34c18d0d39fa5bf9bef22dd8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"1de034bb34c18d0d39fa5bf9bef22dd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.181312 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.181238 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:06.258808 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.258772 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.263447 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.263428 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.281752 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.281730 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:06.382241 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.382212 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:06.482814 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.482748 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:06.583267 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.583237 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found" Jul 06 12:23:06.632068 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.632043 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:23:06.634045 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.634023 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 12:23:06.634149 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.634132 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 12:23:06.672257 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.672236 2572 apiserver.go:52] "Watching apiserver" Jul 06 12:23:06.691018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.690994 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 12:23:06.691018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.691006 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.701193 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.701174 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 12:23:06.701507 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.701489 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gps2r","openshift-network-diagnostics/network-check-target-scpmd","openshift-ovn-kubernetes/ovnkube-node-qnmxh","kube-system/global-pull-secret-syncer-8x6cj","kube-system/konnectivity-agent-gkr2l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96","openshift-dns/node-resolver-fbxl6","openshift-image-registry/node-ca-4xqvf","openshift-multus/multus-additional-cni-plugins-rbfcf","openshift-multus/network-metrics-daemon-f5mqz","openshift-network-operator/iptables-alerter-v6vh5","openshift-cluster-node-tuning-operator/tuned-wpg85"] Jul 06 12:23:06.703006 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.702993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.704082 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.704060 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.705635 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.705621 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:06.706892 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.706878 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.707915 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.707902 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.708285 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.708273 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 12:23:06.708333 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.708325 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" Jul 06 12:23:06.708966 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.708952 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:06.709039 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.709014 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:06.709528 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.709514 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dl5wz\"" Jul 06 12:23:06.709905 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.709890 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 12:23:06.710112 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.710094 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:06.710181 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.710161 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:06.710922 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.710908 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 12:23:06.711183 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.711172 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:06.712574 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.712553 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.712701 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.712637 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 12:23:06.712770 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.712751 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 12:23:06.713625 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.713612 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.714098 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.714081 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 12:23:06.714177 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.714103 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 12:23:06.714177 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.714099 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 12:18:05 +0000 UTC" deadline="2027-12-05 18:37:44.262993484 +0000 UTC" Jul 06 12:23:06.714177 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.714120 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12414h14m37.548876446s" Jul 06 12:23:06.714177 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.714120 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 12:23:06.714909 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.714897 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.716150 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.716134 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"] Jul 06 12:23:06.716225 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.716213 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:06.716292 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.716274 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:06.716923 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.716906 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fzt9w\"" Jul 06 12:23:06.717085 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.717070 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 12:23:06.717217 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.717199 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jdd9q\"" Jul 06 12:23:06.717318 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.717298 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 12:23:06.717402 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.717337 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 12:23:06.717402 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.717395 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 12:23:06.722307 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722292 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 12:23:06.722307 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722299 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 12:23:06.722444 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722335 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 12:23:06.722444 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722352 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 12:23:06.722444 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722361 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 12:23:06.722444 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722306 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 12:23:06.722444 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722358 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 12:23:06.722699 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722455 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ssp9g\"" Jul 06 12:23:06.722699 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722457 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lxntt\"" Jul 06 12:23:06.722890 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.722875 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 12:23:06.723242 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.723228 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 12:23:06.725252 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.725238 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 12:23:06.726607 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.726592 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 12:23:06.726607 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.726600 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 12:23:06.726737 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.726603 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dx277\"" Jul 06 12:23:06.727428 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.727414 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 12:23:06.727501 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.727432 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 12:23:06.727550 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.727485 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k9wb5\"" Jul 06 12:23:06.727609 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.727506 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 12:23:06.727691 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.727677 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 12:23:06.727752 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.727690 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z5q99\"" Jul 06 12:23:06.727860 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.727844 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 12:23:06.728375 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.728361 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j2mm8\"" Jul 06 12:23:06.776983 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.776963 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 12:23:06.777197 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.777009 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"] Jul 06 12:23:06.778572 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.778548 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-67lr2" Jul 06 12:23:06.791368 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.791352 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 12:23:06.797948 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.797907 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-67lr2" Jul 06 12:23:06.797948 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.797932 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdz5\" (UniqueName: \"kubernetes.io/projected/fddefa28-3b87-4803-833e-818c0d54b665-kube-api-access-8zdz5\") pod \"node-resolver-fbxl6\" (UID: \"fddefa28-3b87-4803-833e-818c0d54b665\") " pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.798109 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.797963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5302ffb2-87cb-4aba-8e8a-0e7208e8b217-serviceca\") pod \"node-ca-4xqvf\" (UID: \"5302ffb2-87cb-4aba-8e8a-0e7208e8b217\") " pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.798109 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798002 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-cnibin\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.798109 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798041 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-ovn-node-metrics-cert\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.798109 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798066 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-modprobe-d\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.798109 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798087 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.798109 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798111 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798131 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40839c53-c59a-46b6-8299-4a2ed1c4ec90-dbus\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-run-systemd\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798161 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a-host-slash\") pod \"iptables-alerter-v6vh5\" (UID: \"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a\") " pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798190 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2xgb\" (UniqueName: \"kubernetes.io/projected/2598c5d0-e3bb-4604-bd89-064fcf5e0319-kube-api-access-c2xgb\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798216 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-node-log\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798234 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-sysconfig\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798251 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-sys\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798287 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-lib-modules\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-host\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798338 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-run-netns\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.798397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798386 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-run-ovn\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798439 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-systemd\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798463 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fddefa28-3b87-4803-833e-818c0d54b665-hosts-file\") pod \"node-resolver-fbxl6\" (UID: \"fddefa28-3b87-4803-833e-818c0d54b665\") " pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798485 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddefa28-3b87-4803-833e-818c0d54b665-tmp-dir\") pod \"node-resolver-fbxl6\" (UID: \"fddefa28-3b87-4803-833e-818c0d54b665\") " pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798506 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5302ffb2-87cb-4aba-8e8a-0e7208e8b217-host\") pod \"node-ca-4xqvf\" (UID: \"5302ffb2-87cb-4aba-8e8a-0e7208e8b217\") " pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798529 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-sys-fs\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798560 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-ovnkube-config\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798583 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-sysctl-d\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798606 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-system-cni-dir\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798638 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-run-k8s-cni-cncf-io\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-kubelet\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798768 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-log-socket\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798794 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hj6\" (UniqueName: \"kubernetes.io/projected/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-kube-api-access-64hj6\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-var-lib-cni-multus\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798854 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-etc-kubernetes\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.798988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798878 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798895 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-var-lib-openvswitch\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798922 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-sysctl-conf\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-tuned\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrr6\" (UniqueName: \"kubernetes.io/projected/1e2fddf8-1ba4-488d-839c-05b1fad343a8-kube-api-access-cbrr6\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.798985 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-device-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799022 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a-iptables-alerter-script\") pod \"iptables-alerter-v6vh5\" (UID: \"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a\") " pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799053 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hjs\" (UniqueName: \"kubernetes.io/projected/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-kube-api-access-g8hjs\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799070 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-cni-dir\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799089 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e2fddf8-1ba4-488d-839c-05b1fad343a8-cni-binary-copy\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-socket-dir-parent\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799150 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-var-lib-cni-bin\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799172 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-hostroot\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799204 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7vg\" (UniqueName: \"kubernetes.io/projected/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-kube-api-access-kr7vg\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799233 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799270 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-run-openvswitch\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.799756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799338 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-os-release\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799373 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-run-multus-certs\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799400 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21fedbcf-6e72-41a7-a196-aff0670ba1f3-konnectivity-ca\") pod \"konnectivity-agent-gkr2l\" (UID: \"21fedbcf-6e72-41a7-a196-aff0670ba1f3\") " pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799419 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-registration-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799465 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-systemd-units\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799494 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-slash\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799532 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-cnibin\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799561 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkgl\" (UniqueName: \"kubernetes.io/projected/5302ffb2-87cb-4aba-8e8a-0e7208e8b217-kube-api-access-jtkgl\") pod \"node-ca-4xqvf\" (UID: \"5302ffb2-87cb-4aba-8e8a-0e7208e8b217\") " pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-system-cni-dir\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799634 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799688 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-kubernetes\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799708 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-run\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799726 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-cni-binary-copy\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgmqj\" (UniqueName: \"kubernetes.io/projected/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-kube-api-access-mgmqj\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmq4c\" (UniqueName: \"kubernetes.io/projected/e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a-kube-api-access-qmq4c\") pod \"iptables-alerter-v6vh5\" (UID: \"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a\") " pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:06.800534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799810 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-ovnkube-script-lib\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799863 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-env-overrides\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799901 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-var-lib-kubelet\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799936 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-var-lib-kubelet\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.799973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40839c53-c59a-46b6-8299-4a2ed1c4ec90-kubelet-config\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800005 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800034 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-tmp\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800050 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-conf-dir\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800068 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21fedbcf-6e72-41a7-a196-aff0670ba1f3-agent-certs\") pod \"konnectivity-agent-gkr2l\" (UID: \"21fedbcf-6e72-41a7-a196-aff0670ba1f3\") " pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800091 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-os-release\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800115 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-socket-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-etc-openvswitch\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-cni-netd\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800208 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-daemon-config\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800227 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-cni-bin\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.801241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.800265 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-run-netns\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.868800 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:06.868771 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb0dda8697e373c519f6dc1fe8e110b.slice/crio-c458ae790838137fa116c35dcebc4dc98e1cfba3c929044e5b9b67af3ebd33be WatchSource:0}: Error finding container c458ae790838137fa116c35dcebc4dc98e1cfba3c929044e5b9b67af3ebd33be: Status 404 returned error can't find the container with id c458ae790838137fa116c35dcebc4dc98e1cfba3c929044e5b9b67af3ebd33be Jul 06 12:23:06.869217 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:06.869197 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de034bb34c18d0d39fa5bf9bef22dd8.slice/crio-1244da496d54a005988dd9986a1bcb93ed43f70fc77c3ca90f00f931c6432721 WatchSource:0}: Error finding container 1244da496d54a005988dd9986a1bcb93ed43f70fc77c3ca90f00f931c6432721: Status 404 returned error can't find the container with id 1244da496d54a005988dd9986a1bcb93ed43f70fc77c3ca90f00f931c6432721 Jul 06 12:23:06.875464 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.875447 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 12:23:06.900608 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900583 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkgl\" (UniqueName: \"kubernetes.io/projected/5302ffb2-87cb-4aba-8e8a-0e7208e8b217-kube-api-access-jtkgl\") pod \"node-ca-4xqvf\" (UID: \"5302ffb2-87cb-4aba-8e8a-0e7208e8b217\") " pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.900738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-system-cni-dir\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.900738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.900738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-kubernetes\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.900738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900673 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-run\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.900738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900717 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-cni-binary-copy\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.900738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900727 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-system-cni-dir\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.901001 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900750 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-kubernetes\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.901001 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900733 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgmqj\" (UniqueName: \"kubernetes.io/projected/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-kube-api-access-mgmqj\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.901001 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.900777 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.901139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.901117 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-run\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.901139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.901123 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmq4c\" (UniqueName: \"kubernetes.io/projected/e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a-kube-api-access-qmq4c\") pod \"iptables-alerter-v6vh5\" (UID: \"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a\") " pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:06.901230 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.901169 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-ovnkube-script-lib\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.901230 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.901191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-env-overrides\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.901737 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-var-lib-kubelet\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.901857 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-var-lib-kubelet\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.901891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-var-lib-kubelet\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.901948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40839c53-c59a-46b6-8299-4a2ed1c4ec90-kubelet-config\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.901998 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-env-overrides\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902054 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-var-lib-kubelet\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902095 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-tmp\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-conf-dir\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902165 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-cni-binary-copy\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902103 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-ovnkube-script-lib\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.902189 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902246 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40839c53-c59a-46b6-8299-4a2ed1c4ec90-kubelet-config\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21fedbcf-6e72-41a7-a196-aff0670ba1f3-agent-certs\") pod \"konnectivity-agent-gkr2l\" (UID: \"21fedbcf-6e72-41a7-a196-aff0670ba1f3\") " pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.902297 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs podName:2598c5d0-e3bb-4604-bd89-064fcf5e0319 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:07.402259852 +0000 UTC m=+3.480807870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs") pod "network-metrics-daemon-f5mqz" (UID: "2598c5d0-e3bb-4604-bd89-064fcf5e0319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902338 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-conf-dir\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.902690 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902378 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-os-release\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-socket-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902463 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902507 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-etc-openvswitch\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902510 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-os-release\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902534 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-cni-netd\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902579 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-daemon-config\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902585 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-cni-netd\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902580 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-socket-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902639 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-etc-openvswitch\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.902650 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.902732 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret podName:40839c53-c59a-46b6-8299-4a2ed1c4ec90 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:07.402715695 +0000 UTC m=+3.481263699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret") pod "global-pull-secret-syncer-8x6cj" (UID: "40839c53-c59a-46b6-8299-4a2ed1c4ec90") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-cni-bin\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-run-netns\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902840 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdz5\" (UniqueName: \"kubernetes.io/projected/fddefa28-3b87-4803-833e-818c0d54b665-kube-api-access-8zdz5\") pod \"node-resolver-fbxl6\" (UID: \"fddefa28-3b87-4803-833e-818c0d54b665\") " pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902868 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5302ffb2-87cb-4aba-8e8a-0e7208e8b217-serviceca\") pod \"node-ca-4xqvf\" (UID: \"5302ffb2-87cb-4aba-8e8a-0e7208e8b217\") " pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.903680 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-cnibin\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-ovn-node-metrics-cert\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.902982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-modprobe-d\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903012 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903043 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903072 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40839c53-c59a-46b6-8299-4a2ed1c4ec90-dbus\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-run-systemd\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903125 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a-host-slash\") pod \"iptables-alerter-v6vh5\" (UID: \"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a\") " pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903153 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2xgb\" (UniqueName: \"kubernetes.io/projected/2598c5d0-e3bb-4604-bd89-064fcf5e0319-kube-api-access-c2xgb\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903179 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-node-log\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903209 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-sysconfig\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903231 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-sys\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903237 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-daemon-config\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903259 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-lib-modules\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903338 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-run-systemd\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903341 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-run-netns\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903406 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-cnibin\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.904540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903531 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-sysconfig\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903594 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a-host-slash\") pod \"iptables-alerter-v6vh5\" (UID: \"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a\") " pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903612 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-sys\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903683 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-node-log\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903741 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-modprobe-d\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.903914 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-lib-modules\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904172 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-cni-bin\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-host\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904513 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904548 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-run-netns\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904572 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-run-ovn\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904603 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904633 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-systemd\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904681 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fddefa28-3b87-4803-833e-818c0d54b665-hosts-file\") pod \"node-resolver-fbxl6\" (UID: \"fddefa28-3b87-4803-833e-818c0d54b665\") " pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904709 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddefa28-3b87-4803-833e-818c0d54b665-tmp-dir\") pod \"node-resolver-fbxl6\" (UID: \"fddefa28-3b87-4803-833e-818c0d54b665\") " pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5302ffb2-87cb-4aba-8e8a-0e7208e8b217-host\") pod \"node-ca-4xqvf\" (UID: \"5302ffb2-87cb-4aba-8e8a-0e7208e8b217\") " pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904786 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-run-ovn\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904792 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-sys-fs\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.905345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904815 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904832 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5302ffb2-87cb-4aba-8e8a-0e7208e8b217-host\") pod \"node-ca-4xqvf\" (UID: \"5302ffb2-87cb-4aba-8e8a-0e7208e8b217\") " pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-ovnkube-config\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.904914 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fddefa28-3b87-4803-833e-818c0d54b665-hosts-file\") pod \"node-resolver-fbxl6\" (UID: \"fddefa28-3b87-4803-833e-818c0d54b665\") " pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905023 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-sys-fs\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905104 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-systemd\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905109 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddefa28-3b87-4803-833e-818c0d54b665-tmp-dir\") pod \"node-resolver-fbxl6\" (UID: \"fddefa28-3b87-4803-833e-818c0d54b665\") " pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905170 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905184 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-sysctl-d\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-system-cni-dir\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-run-k8s-cni-cncf-io\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-kubelet\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-log-socket\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905317 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40839c53-c59a-46b6-8299-4a2ed1c4ec90-dbus\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905330 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hj6\" (UniqueName: \"kubernetes.io/projected/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-kube-api-access-64hj6\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-var-lib-cni-multus\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905365 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-etc-kubernetes\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.906214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905442 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-run-netns\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905524 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-sysctl-d\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905562 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905595 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-var-lib-openvswitch\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905620 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-sysctl-conf\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905673 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-tuned\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrr6\" (UniqueName: \"kubernetes.io/projected/1e2fddf8-1ba4-488d-839c-05b1fad343a8-kube-api-access-cbrr6\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-device-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905763 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a-iptables-alerter-script\") pod \"iptables-alerter-v6vh5\" (UID: \"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a\") " pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905868 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-log-socket\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905890 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-system-cni-dir\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.905969 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-host\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906028 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-etc-kubernetes\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906038 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-var-lib-cni-multus\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906086 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hjs\" (UniqueName: \"kubernetes.io/projected/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-kube-api-access-g8hjs\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-cni-dir\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906186 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e2fddf8-1ba4-488d-839c-05b1fad343a8-cni-binary-copy\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-socket-dir-parent\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906249 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-sysctl-conf\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906253 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-var-lib-cni-bin\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-hostroot\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906298 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-var-lib-cni-bin\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906358 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-kubelet\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906407 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-cni-dir\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906419 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-device-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906451 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7vg\" (UniqueName: \"kubernetes.io/projected/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-kube-api-access-kr7vg\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906495 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-var-lib-openvswitch\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906546 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906606 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-multus-socket-dir-parent\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906692 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906847 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906890 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-run-openvswitch\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.906942 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e2fddf8-1ba4-488d-839c-05b1fad343a8-cni-binary-copy\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907003 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-os-release\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.907950 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907035 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907044 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-run-multus-certs\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907063 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-ovnkube-config\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21fedbcf-6e72-41a7-a196-aff0670ba1f3-konnectivity-ca\") pod \"konnectivity-agent-gkr2l\" (UID: \"21fedbcf-6e72-41a7-a196-aff0670ba1f3\") " pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907085 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a-iptables-alerter-script\") pod \"iptables-alerter-v6vh5\" (UID: \"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a\") " pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907148 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-run-openvswitch\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907153 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-registration-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907139 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-run-k8s-cni-cncf-io\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907195 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-systemd-units\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-slash\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-cnibin\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907281 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-ovn-node-metrics-cert\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907292 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21fedbcf-6e72-41a7-a196-aff0670ba1f3-agent-certs\") pod \"konnectivity-agent-gkr2l\" (UID: \"21fedbcf-6e72-41a7-a196-aff0670ba1f3\") " pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907315 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-systemd-units\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907344 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-hostroot\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907428 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-host-slash\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907803 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-os-release\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907377 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-registration-dir\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.908524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907831 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-host-run-multus-certs\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.909069 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907818 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e2fddf8-1ba4-488d-839c-05b1fad343a8-cnibin\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.909069 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.907978 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5302ffb2-87cb-4aba-8e8a-0e7208e8b217-serviceca\") pod \"node-ca-4xqvf\" (UID: \"5302ffb2-87cb-4aba-8e8a-0e7208e8b217\") " pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.909069 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.908231 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-tmp\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.909069 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.908771 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21fedbcf-6e72-41a7-a196-aff0670ba1f3-konnectivity-ca\") pod \"konnectivity-agent-gkr2l\" (UID: \"21fedbcf-6e72-41a7-a196-aff0670ba1f3\") " pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:06.909069 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.908960 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-etc-tuned\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.929259 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.929241 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdz5\" (UniqueName: \"kubernetes.io/projected/fddefa28-3b87-4803-833e-818c0d54b665-kube-api-access-8zdz5\") pod \"node-resolver-fbxl6\" (UID: \"fddefa28-3b87-4803-833e-818c0d54b665\") " pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:06.929789 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.929771 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:23:06.929789 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.929791 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:23:06.929894 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.929801 2572 projected.go:196] Error preparing data for projected volume kube-api-access-rn55r for pod openshift-network-diagnostics/network-check-target-scpmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:06.929894 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:06.929857 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r podName:acffbace-3fd7-4848-8a5b-3b24904b6c1b nodeName:}" failed. No retries permitted until 2026-07-06 12:23:07.429842754 +0000 UTC m=+3.508390787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rn55r" (UniqueName: "kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r") pod "network-check-target-scpmd" (UID: "acffbace-3fd7-4848-8a5b-3b24904b6c1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:06.930703 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.930684 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7vg\" (UniqueName: \"kubernetes.io/projected/2788ceaa-8992-4990-9bf3-abcb9cc9d57f-kube-api-access-kr7vg\") pod \"multus-additional-cni-plugins-rbfcf\" (UID: \"2788ceaa-8992-4990-9bf3-abcb9cc9d57f\") " pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:06.930935 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.930919 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgmqj\" (UniqueName: \"kubernetes.io/projected/8c8bb9d6-7ec8-495f-a597-4c32a895fd0a-kube-api-access-mgmqj\") pod \"aws-ebs-csi-driver-node-8mh96\" (UID: \"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:06.932650 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.932627 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hjs\" (UniqueName: \"kubernetes.io/projected/64a4bf7d-026e-44d8-bbed-e9c6a2afca09-kube-api-access-g8hjs\") pod \"ovnkube-node-qnmxh\" (UID: \"64a4bf7d-026e-44d8-bbed-e9c6a2afca09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:06.934861 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.934840 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrr6\" (UniqueName: \"kubernetes.io/projected/1e2fddf8-1ba4-488d-839c-05b1fad343a8-kube-api-access-cbrr6\") pod \"multus-gps2r\" (UID: \"1e2fddf8-1ba4-488d-839c-05b1fad343a8\") " pod="openshift-multus/multus-gps2r" Jul 06 12:23:06.935384 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.935365 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkgl\" (UniqueName: \"kubernetes.io/projected/5302ffb2-87cb-4aba-8e8a-0e7208e8b217-kube-api-access-jtkgl\") pod \"node-ca-4xqvf\" (UID: \"5302ffb2-87cb-4aba-8e8a-0e7208e8b217\") " pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:06.935563 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.935546 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2xgb\" (UniqueName: \"kubernetes.io/projected/2598c5d0-e3bb-4604-bd89-064fcf5e0319-kube-api-access-c2xgb\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:06.935642 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.935627 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hj6\" (UniqueName: \"kubernetes.io/projected/b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8-kube-api-access-64hj6\") pod \"tuned-wpg85\" (UID: \"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:06.938816 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:06.938800 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmq4c\" (UniqueName: \"kubernetes.io/projected/e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a-kube-api-access-qmq4c\") pod \"iptables-alerter-v6vh5\" (UID: \"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a\") " pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:07.016251 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.016186 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:23:07.035344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.035321 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:07.042113 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.042098 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:23:07.043906 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:07.043878 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a4bf7d_026e_44d8_bbed_e9c6a2afca09.slice/crio-2963cc3a58c11054797142f4b343205612c4c5f83512c747ef81fec36ca63bac WatchSource:0}: Error finding container 2963cc3a58c11054797142f4b343205612c4c5f83512c747ef81fec36ca63bac: Status 404 returned error can't find the container with id 2963cc3a58c11054797142f4b343205612c4c5f83512c747ef81fec36ca63bac Jul 06 12:23:07.054947 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.054922 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" Jul 06 12:23:07.061898 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:07.061876 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8bb9d6_7ec8_495f_a597_4c32a895fd0a.slice/crio-b27c20d3352920c58eac8efaabf8dc99fbe79db7ca924564214562b259935626 WatchSource:0}: Error finding container b27c20d3352920c58eac8efaabf8dc99fbe79db7ca924564214562b259935626: Status 404 returned error can't find the container with id b27c20d3352920c58eac8efaabf8dc99fbe79db7ca924564214562b259935626 Jul 06 12:23:07.063446 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.063428 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v6vh5" Jul 06 12:23:07.070963 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:07.070945 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61d43cd_14c9_4e80_83ae_9b8b9ef80c5a.slice/crio-0eec5aa72c95e2992314059f2a890e3cbbd2a619aacbb65333dffa37aa7f4a51 WatchSource:0}: Error finding container 0eec5aa72c95e2992314059f2a890e3cbbd2a619aacbb65333dffa37aa7f4a51: Status 404 returned error can't find the container with id 0eec5aa72c95e2992314059f2a890e3cbbd2a619aacbb65333dffa37aa7f4a51 Jul 06 12:23:07.083140 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.083121 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wpg85" Jul 06 12:23:07.089073 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:07.089054 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b4bb8a_45f8_4c65_99eb_aef19e80b3e8.slice/crio-6d79fd8ed715a117cb250962edc28aee902478fcad631feaea666723c9241181 WatchSource:0}: Error finding container 6d79fd8ed715a117cb250962edc28aee902478fcad631feaea666723c9241181: Status 404 returned error can't find the container with id 6d79fd8ed715a117cb250962edc28aee902478fcad631feaea666723c9241181 Jul 06 12:23:07.105110 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.105086 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gps2r" Jul 06 12:23:07.112276 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:07.112252 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2fddf8_1ba4_488d_839c_05b1fad343a8.slice/crio-ab05445a1d4ba6d11c8c5daaf1c00c8d1d2b3282f55f349992617fb1e52bec54 WatchSource:0}: Error finding container ab05445a1d4ba6d11c8c5daaf1c00c8d1d2b3282f55f349992617fb1e52bec54: Status 404 returned error can't find the container with id ab05445a1d4ba6d11c8c5daaf1c00c8d1d2b3282f55f349992617fb1e52bec54 Jul 06 12:23:07.131210 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.131190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:07.137907 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:07.137879 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fedbcf_6e72_41a7_a196_aff0670ba1f3.slice/crio-a14fdf18074dc321c57121dad5896fce1968db21c111d51f008baf70494126c8 WatchSource:0}: Error finding container a14fdf18074dc321c57121dad5896fce1968db21c111d51f008baf70494126c8: Status 404 returned error can't find the container with id a14fdf18074dc321c57121dad5896fce1968db21c111d51f008baf70494126c8 Jul 06 12:23:07.149156 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.149140 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fbxl6" Jul 06 12:23:07.154798 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.154776 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xqvf" Jul 06 12:23:07.155958 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:07.155939 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddefa28_3b87_4803_833e_818c0d54b665.slice/crio-a56f68bda01059c3f6b9817c1014b9ff9a82b581490f82e21b9f178cead57031 WatchSource:0}: Error finding container a56f68bda01059c3f6b9817c1014b9ff9a82b581490f82e21b9f178cead57031: Status 404 returned error can't find the container with id a56f68bda01059c3f6b9817c1014b9ff9a82b581490f82e21b9f178cead57031 Jul 06 12:23:07.161154 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.161135 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" Jul 06 12:23:07.161984 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:07.161965 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5302ffb2_87cb_4aba_8e8a_0e7208e8b217.slice/crio-81bffe70bf9b0b7e9ff29826480df1e03cf48a1a701d50f9a3eeebc8e9ded6da WatchSource:0}: Error finding container 81bffe70bf9b0b7e9ff29826480df1e03cf48a1a701d50f9a3eeebc8e9ded6da: Status 404 returned error can't find the container with id 81bffe70bf9b0b7e9ff29826480df1e03cf48a1a701d50f9a3eeebc8e9ded6da Jul 06 12:23:07.168726 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:07.168704 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2788ceaa_8992_4990_9bf3_abcb9cc9d57f.slice/crio-9f4c5bdc2b1f828611edc6fbe146a3efb5bf8913ad837a474ff8dba3da0528ee WatchSource:0}: Error finding container 9f4c5bdc2b1f828611edc6fbe146a3efb5bf8913ad837a474ff8dba3da0528ee: Status 404 returned error can't find the container with id 9f4c5bdc2b1f828611edc6fbe146a3efb5bf8913ad837a474ff8dba3da0528ee Jul 06 12:23:07.411018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.410938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:07.411018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.410991 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:07.411223 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:07.411100 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:07.411223 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:07.411155 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:07.411223 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:07.411177 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs podName:2598c5d0-e3bb-4604-bd89-064fcf5e0319 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:08.411154467 +0000 UTC m=+4.489702477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs") pod "network-metrics-daemon-f5mqz" (UID: "2598c5d0-e3bb-4604-bd89-064fcf5e0319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:07.411223 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:07.411203 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret podName:40839c53-c59a-46b6-8299-4a2ed1c4ec90 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:08.411188765 +0000 UTC m=+4.489736775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret") pod "global-pull-secret-syncer-8x6cj" (UID: "40839c53-c59a-46b6-8299-4a2ed1c4ec90") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:07.511706 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.511670 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:07.511869 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:07.511797 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:23:07.511869 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:07.511820 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:23:07.511869 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:07.511834 2572 projected.go:196] Error preparing data for projected volume kube-api-access-rn55r for pod openshift-network-diagnostics/network-check-target-scpmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:07.512032 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:07.511887 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r podName:acffbace-3fd7-4848-8a5b-3b24904b6c1b nodeName:}" failed. No retries permitted until 2026-07-06 12:23:08.511869768 +0000 UTC m=+4.590417787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rn55r" (UniqueName: "kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r") pod "network-check-target-scpmd" (UID: "acffbace-3fd7-4848-8a5b-3b24904b6c1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:07.674329 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.674249 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:23:07.800849 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.800690 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 12:18:06 +0000 UTC" deadline="2028-03-18 22:04:57.302986896 +0000 UTC" Jul 06 12:23:07.800849 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.800727 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14913h41m49.502264307s" Jul 06 12:23:07.826925 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.826901 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:07.827095 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:07.827066 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:07.852015 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.851887 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gps2r" event={"ID":"1e2fddf8-1ba4-488d-839c-05b1fad343a8","Type":"ContainerStarted","Data":"ab05445a1d4ba6d11c8c5daaf1c00c8d1d2b3282f55f349992617fb1e52bec54"} Jul 06 12:23:07.868874 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.868840 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" event={"ID":"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a","Type":"ContainerStarted","Data":"b27c20d3352920c58eac8efaabf8dc99fbe79db7ca924564214562b259935626"} Jul 06 12:23:07.887280 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.887216 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" event={"ID":"9bb0dda8697e373c519f6dc1fe8e110b","Type":"ContainerStarted","Data":"c458ae790838137fa116c35dcebc4dc98e1cfba3c929044e5b9b67af3ebd33be"} Jul 06 12:23:07.915464 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.915429 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" event={"ID":"1de034bb34c18d0d39fa5bf9bef22dd8","Type":"ContainerStarted","Data":"1244da496d54a005988dd9986a1bcb93ed43f70fc77c3ca90f00f931c6432721"} Jul 06 12:23:07.939884 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.939775 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xqvf" event={"ID":"5302ffb2-87cb-4aba-8e8a-0e7208e8b217","Type":"ContainerStarted","Data":"81bffe70bf9b0b7e9ff29826480df1e03cf48a1a701d50f9a3eeebc8e9ded6da"} Jul 06 12:23:07.943929 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.943866 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fbxl6" event={"ID":"fddefa28-3b87-4803-833e-818c0d54b665","Type":"ContainerStarted","Data":"a56f68bda01059c3f6b9817c1014b9ff9a82b581490f82e21b9f178cead57031"} Jul 06 12:23:07.956165 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.956137 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gkr2l" event={"ID":"21fedbcf-6e72-41a7-a196-aff0670ba1f3","Type":"ContainerStarted","Data":"a14fdf18074dc321c57121dad5896fce1968db21c111d51f008baf70494126c8"} Jul 06 12:23:07.965487 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.965458 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wpg85" event={"ID":"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8","Type":"ContainerStarted","Data":"6d79fd8ed715a117cb250962edc28aee902478fcad631feaea666723c9241181"} Jul 06 12:23:07.975052 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.975001 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v6vh5" event={"ID":"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a","Type":"ContainerStarted","Data":"0eec5aa72c95e2992314059f2a890e3cbbd2a619aacbb65333dffa37aa7f4a51"} Jul 06 12:23:07.995883 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:07.995854 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" event={"ID":"64a4bf7d-026e-44d8-bbed-e9c6a2afca09","Type":"ContainerStarted","Data":"2963cc3a58c11054797142f4b343205612c4c5f83512c747ef81fec36ca63bac"} Jul 06 12:23:08.004990 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:08.004866 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" event={"ID":"2788ceaa-8992-4990-9bf3-abcb9cc9d57f","Type":"ContainerStarted","Data":"9f4c5bdc2b1f828611edc6fbe146a3efb5bf8913ad837a474ff8dba3da0528ee"} Jul 06 12:23:08.418905 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:08.418823 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:08.418905 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:08.418873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:08.419099 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.418987 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:08.419099 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.419047 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret podName:40839c53-c59a-46b6-8299-4a2ed1c4ec90 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:10.419029965 +0000 UTC m=+6.497577970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret") pod "global-pull-secret-syncer-8x6cj" (UID: "40839c53-c59a-46b6-8299-4a2ed1c4ec90") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:08.419477 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.419456 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:08.419595 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.419512 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs podName:2598c5d0-e3bb-4604-bd89-064fcf5e0319 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:10.419496413 +0000 UTC m=+6.498044433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs") pod "network-metrics-daemon-f5mqz" (UID: "2598c5d0-e3bb-4604-bd89-064fcf5e0319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:08.519691 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:08.519631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:08.519895 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.519866 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:23:08.519895 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.519895 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:23:08.520022 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.519908 2572 projected.go:196] Error preparing data for projected volume kube-api-access-rn55r for pod openshift-network-diagnostics/network-check-target-scpmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:08.520022 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.519973 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r podName:acffbace-3fd7-4848-8a5b-3b24904b6c1b nodeName:}" failed. No retries permitted until 2026-07-06 12:23:10.519954889 +0000 UTC m=+6.598502917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rn55r" (UniqueName: "kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r") pod "network-check-target-scpmd" (UID: "acffbace-3fd7-4848-8a5b-3b24904b6c1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:08.801988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:08.801805 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 12:18:06 +0000 UTC" deadline="2028-02-22 23:57:41.404135142 +0000 UTC" Jul 06 12:23:08.801988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:08.801897 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14315h34m32.602242536s" Jul 06 12:23:08.826364 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:08.825743 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:08.826364 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.825858 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:08.826364 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:08.826239 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:08.826364 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:08.826331 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:09.827229 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:09.827195 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:09.827733 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:09.827329 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:10.438213 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:10.438176 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:10.438387 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:10.438229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:10.438458 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.438397 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:10.438520 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.438460 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret podName:40839c53-c59a-46b6-8299-4a2ed1c4ec90 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:14.438442158 +0000 UTC m=+10.516990166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret") pod "global-pull-secret-syncer-8x6cj" (UID: "40839c53-c59a-46b6-8299-4a2ed1c4ec90") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:10.438894 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.438874 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:10.438999 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.438929 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs podName:2598c5d0-e3bb-4604-bd89-064fcf5e0319 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:14.438914581 +0000 UTC m=+10.517462589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs") pod "network-metrics-daemon-f5mqz" (UID: "2598c5d0-e3bb-4604-bd89-064fcf5e0319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:10.538854 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:10.538810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:10.539030 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.538993 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:23:10.539030 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.539015 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:23:10.539030 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.539029 2572 projected.go:196] Error preparing data for projected volume kube-api-access-rn55r for pod openshift-network-diagnostics/network-check-target-scpmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:10.539184 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.539084 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r podName:acffbace-3fd7-4848-8a5b-3b24904b6c1b nodeName:}" failed. No retries permitted until 2026-07-06 12:23:14.539066293 +0000 UTC m=+10.617614318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rn55r" (UniqueName: "kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r") pod "network-check-target-scpmd" (UID: "acffbace-3fd7-4848-8a5b-3b24904b6c1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:10.826841 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:10.826176 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:10.826841 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.826295 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:10.826841 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:10.826347 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:10.826841 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:10.826412 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:11.826008 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:11.825504 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:11.826008 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:11.825642 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:12.826093 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:12.826054 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:12.826505 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:12.826208 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:12.826505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:12.826277 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:12.826505 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:12.826338 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:13.826547 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:13.826512 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:13.826958 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:13.826685 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:14.471282 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:14.471248 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:14.471470 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:14.471298 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:14.471470 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.471452 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:14.471595 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.471514 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret podName:40839c53-c59a-46b6-8299-4a2ed1c4ec90 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:22.471494498 +0000 UTC m=+18.550042508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret") pod "global-pull-secret-syncer-8x6cj" (UID: "40839c53-c59a-46b6-8299-4a2ed1c4ec90") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:14.471879 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.471783 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:14.471879 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.471851 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs podName:2598c5d0-e3bb-4604-bd89-064fcf5e0319 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:22.471833958 +0000 UTC m=+18.550381978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs") pod "network-metrics-daemon-f5mqz" (UID: "2598c5d0-e3bb-4604-bd89-064fcf5e0319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:14.572458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:14.572417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:14.572623 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.572590 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:23:14.572623 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.572613 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:23:14.572766 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.572627 2572 projected.go:196] Error preparing data for projected volume kube-api-access-rn55r for pod openshift-network-diagnostics/network-check-target-scpmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:14.572766 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.572704 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r podName:acffbace-3fd7-4848-8a5b-3b24904b6c1b nodeName:}" failed. No retries permitted until 2026-07-06 12:23:22.572685445 +0000 UTC m=+18.651233462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rn55r" (UniqueName: "kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r") pod "network-check-target-scpmd" (UID: "acffbace-3fd7-4848-8a5b-3b24904b6c1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:14.825519 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:14.825440 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:14.825519 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:14.825467 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:14.825770 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.825572 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:14.825770 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:14.825715 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:15.826397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:15.826348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:15.826812 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:15.826507 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:16.825566 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:16.825537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:16.825763 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:16.825537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:16.825763 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:16.825677 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:16.825865 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:16.825774 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:17.825885 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:17.825738 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:17.825885 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:17.825857 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:18.825692 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:18.825646 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:18.825872 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:18.825646 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:18.825872 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:18.825789 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:18.825872 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:18.825841 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:19.825653 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:19.825615 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:19.825823 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:19.825787 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:20.826113 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:20.826079 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:20.826470 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:20.826080 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:20.826470 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:20.826193 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:20.826470 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:20.826261 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:21.825597 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:21.825558 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:21.825811 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:21.825725 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:22.533691 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:22.533645 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:22.534199 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:22.533704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:22.534199 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.533837 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:22.534199 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.533917 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs podName:2598c5d0-e3bb-4604-bd89-064fcf5e0319 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:38.533895935 +0000 UTC m=+34.612443953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs") pod "network-metrics-daemon-f5mqz" (UID: "2598c5d0-e3bb-4604-bd89-064fcf5e0319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:22.534199 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.533939 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:22.534199 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.534005 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret podName:40839c53-c59a-46b6-8299-4a2ed1c4ec90 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:38.53398636 +0000 UTC m=+34.612534368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret") pod "global-pull-secret-syncer-8x6cj" (UID: "40839c53-c59a-46b6-8299-4a2ed1c4ec90") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:22.634176 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:22.634145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:22.634370 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.634285 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:23:22.634370 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.634304 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:23:22.634370 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.634316 2572 projected.go:196] Error preparing data for projected volume kube-api-access-rn55r for pod openshift-network-diagnostics/network-check-target-scpmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:22.634519 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.634376 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r podName:acffbace-3fd7-4848-8a5b-3b24904b6c1b nodeName:}" failed. No retries permitted until 2026-07-06 12:23:38.634357338 +0000 UTC m=+34.712905366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rn55r" (UniqueName: "kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r") pod "network-check-target-scpmd" (UID: "acffbace-3fd7-4848-8a5b-3b24904b6c1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:22.826128 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:22.826050 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:22.826291 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:22.826058 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:22.826291 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.826157 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:22.826291 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:22.826249 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:23.826195 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:23.826152 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:23.826557 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:23.826303 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:24.825498 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:24.825462 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:24.825697 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:24.825514 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:24.825697 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:24.825621 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:24.825827 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:24.825725 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:25.826158 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:25.826120 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:25.826604 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:25.826233 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:26.825250 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:26.825228 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:26.825364 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:26.825346 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:26.825420 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:26.825397 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:26.825517 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:26.825500 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:27.065967 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:27.065633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wpg85" event={"ID":"b9b4bb8a-45f8-4c65-99eb-aef19e80b3e8","Type":"ContainerStarted","Data":"a332b9a64fef42449b040a75f8c421bf99a34d8c573903ee15d5e79b520f2359"} Jul 06 12:23:27.076596 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:27.074650 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" event={"ID":"64a4bf7d-026e-44d8-bbed-e9c6a2afca09","Type":"ContainerStarted","Data":"98861f08182260af1bb12438e065436192fd54383a73ac1805154eb2b6ef2b90"} Jul 06 12:23:27.076596 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:27.074941 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" event={"ID":"64a4bf7d-026e-44d8-bbed-e9c6a2afca09","Type":"ContainerStarted","Data":"b98123c5de8abcbb7acc28807616710584cafdb0d02759aeaaf18efc79773557"} Jul 06 12:23:27.088700 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:27.088638 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gps2r" event={"ID":"1e2fddf8-1ba4-488d-839c-05b1fad343a8","Type":"ContainerStarted","Data":"65ba73c605cc0f2c5bd2247f87c5f968370e50ee1772a55bbe57f5a6c2ee7b7e"} Jul 06 12:23:27.090875 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:27.090822 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" event={"ID":"9bb0dda8697e373c519f6dc1fe8e110b","Type":"ContainerStarted","Data":"10f0e10be824354549171faf4db391bece4a114e162cf36651c5e471b81b26f7"} Jul 06 12:23:27.100732 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:27.100646 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wpg85" podStartSLOduration=2.408302244 podStartE2EDuration="22.100626053s" totalImagesPullingTime="19.692323809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:27.100353048 +0000 UTC m=+23.178901078" watchObservedRunningTime="2026-07-06 12:23:27.100626053 +0000 UTC m=+23.179174081" Jul 06 12:23:27.119219 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:27.119161 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" podStartSLOduration=21.119141736 podStartE2EDuration="21.119141736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:27.118822371 +0000 UTC m=+23.197370397" watchObservedRunningTime="2026-07-06 12:23:27.119141736 +0000 UTC m=+23.197689971" Jul 06 12:23:27.825829 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:27.825756 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:27.825977 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:27.825908 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:28.094698 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.094616 2572 generic.go:356] "Generic (PLEG): container finished" podID="2788ceaa-8992-4990-9bf3-abcb9cc9d57f" containerID="6ff91d8dc04588ef1b5b1c6c1cae7c9ee444d43446006aa239b65288ab4c2756" exitCode=0 Jul 06 12:23:28.095467 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.094710 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" event={"ID":"2788ceaa-8992-4990-9bf3-abcb9cc9d57f","Type":"ContainerDied","Data":"6ff91d8dc04588ef1b5b1c6c1cae7c9ee444d43446006aa239b65288ab4c2756"} Jul 06 12:23:28.096343 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.096310 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" event={"ID":"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a","Type":"ContainerStarted","Data":"d98a6f02325038acb4a7fb6fde5e8bf780d9940c7cb0ed7a6a1b9ac43871cac7"} Jul 06 12:23:28.098111 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.098087 2572 generic.go:356] "Generic (PLEG): container finished" podID="1de034bb34c18d0d39fa5bf9bef22dd8" containerID="5b25320f1275f25f8d89f26414dade913896d5d27b3c1b499da45fceb41befd3" exitCode=0 Jul 06 12:23:28.098199 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.098164 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" event={"ID":"1de034bb34c18d0d39fa5bf9bef22dd8","Type":"ContainerDied","Data":"5b25320f1275f25f8d89f26414dade913896d5d27b3c1b499da45fceb41befd3"} Jul 06 12:23:28.099720 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.099693 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xqvf" event={"ID":"5302ffb2-87cb-4aba-8e8a-0e7208e8b217","Type":"ContainerStarted","Data":"5d2c3de6c4dd7c93cbb91451ec73c80a16c8e41140b70d281d7e74a666730dbb"} Jul 06 12:23:28.102811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.102773 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fbxl6" event={"ID":"fddefa28-3b87-4803-833e-818c0d54b665","Type":"ContainerStarted","Data":"cba55c7e110643d2cc0ef643219d0a3afd010c94da82f070d7f966a87b5fa10a"} Jul 06 12:23:28.104250 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.104232 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gkr2l" event={"ID":"21fedbcf-6e72-41a7-a196-aff0670ba1f3","Type":"ContainerStarted","Data":"533bc43557a2f56d94d8847c1a09d5b547172d50ff9836609269e64474273a4d"} Jul 06 12:23:28.106838 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.106816 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" event={"ID":"64a4bf7d-026e-44d8-bbed-e9c6a2afca09","Type":"ContainerStarted","Data":"a0676c8e97e663320b6805b0de1eef0b273f62ab6290a1fbe06493bb9fa45ca6"} Jul 06 12:23:28.106933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.106883 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" event={"ID":"64a4bf7d-026e-44d8-bbed-e9c6a2afca09","Type":"ContainerStarted","Data":"4cc2ac38d04d32c4b40c890c3271edfb5f67861a024dbcccdfadc1a3be9f7219"} Jul 06 12:23:28.106933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.106914 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" event={"ID":"64a4bf7d-026e-44d8-bbed-e9c6a2afca09","Type":"ContainerStarted","Data":"c6f91be42adfbee717914685408b0eb2ef6e90d118b98a917f0471cc8f8fdfe5"} Jul 06 12:23:28.107022 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.106940 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" event={"ID":"64a4bf7d-026e-44d8-bbed-e9c6a2afca09","Type":"ContainerStarted","Data":"21ddd5f3b1b0c3a0f9a48306425d9aed222e30e79e8624557d66e4ec62528205"} Jul 06 12:23:28.130638 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.130604 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gps2r" podStartSLOduration=2.350381601 podStartE2EDuration="22.130593567s" totalImagesPullingTime="19.780211966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:27.145841431 +0000 UTC m=+23.224389461" watchObservedRunningTime="2026-07-06 12:23:28.130593567 +0000 UTC m=+24.209141618" Jul 06 12:23:28.149706 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.149465 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4xqvf" podStartSLOduration=2.530164024 podStartE2EDuration="22.149449463s" totalImagesPullingTime="19.619285439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:28.148849772 +0000 UTC m=+24.227397800" watchObservedRunningTime="2026-07-06 12:23:28.149449463 +0000 UTC m=+24.227997492" Jul 06 12:23:28.221694 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.221642 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gkr2l" podStartSLOduration=7.414027078 podStartE2EDuration="22.221625274s" totalImagesPullingTime="14.807598196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:28.199626974 +0000 UTC m=+24.278175001" watchObservedRunningTime="2026-07-06 12:23:28.221625274 +0000 UTC m=+24.300173300" Jul 06 12:23:28.221845 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.221818 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fbxl6" podStartSLOduration=2.595394668 podStartE2EDuration="22.221809086s" totalImagesPullingTime="19.626414418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:28.221502722 +0000 UTC m=+24.300050753" watchObservedRunningTime="2026-07-06 12:23:28.221809086 +0000 UTC m=+24.300357113" Jul 06 12:23:28.327557 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.327516 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:28.359566 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.359547 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 12:23:28.400121 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.400098 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:28.755490 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.755340 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T12:23:28.359561272Z","UUID":"8ca2e042-6e76-480d-a013-5926fa181938","Handler":null,"Name":"","Endpoint":""} Jul 06 12:23:28.757260 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.757220 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 12:23:28.757260 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.757266 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 12:23:28.825809 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.825783 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:28.825982 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:28.825888 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:28.825982 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:28.825953 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:28.826095 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:28.826048 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:29.111946 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:29.111855 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" event={"ID":"1de034bb34c18d0d39fa5bf9bef22dd8","Type":"ContainerStarted","Data":"b291e59ff55d1add1b35915359c8d03e60206aed462b43705fe49870fd14c378"} Jul 06 12:23:29.113725 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:29.113690 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v6vh5" event={"ID":"e61d43cd-14c9-4e80-83ae-9b8b9ef80c5a","Type":"ContainerStarted","Data":"4193e3cb044c8e31bb94339dc7da0c3ceb22296aa14008952b949b1afc4b783f"} Jul 06 12:23:29.115842 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:29.115803 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" event={"ID":"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a","Type":"ContainerStarted","Data":"bcd3338cc983d774ca0e867b39b42262aee748759d1f3a2d05c7142579cf3587"} Jul 06 12:23:29.139753 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:29.139713 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" podStartSLOduration=23.139701056 podStartE2EDuration="23.139701056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:29.139503783 +0000 UTC m=+25.218051812" watchObservedRunningTime="2026-07-06 12:23:29.139701056 +0000 UTC m=+25.218249082" Jul 06 12:23:29.157930 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:29.157881 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v6vh5" podStartSLOduration=4.476167263 podStartE2EDuration="24.157868172s" totalImagesPullingTime="19.681700909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:29.157615084 +0000 UTC m=+25.236163127" watchObservedRunningTime="2026-07-06 12:23:29.157868172 +0000 UTC m=+25.236416198" Jul 06 12:23:29.285748 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:29.285712 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:29.424571 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:29.424495 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:29.826340 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:29.826312 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:29.826495 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:29.826454 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:30.123708 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:30.123612 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" event={"ID":"64a4bf7d-026e-44d8-bbed-e9c6a2afca09","Type":"ContainerStarted","Data":"2d4f802abaa25dde5c25015455ab4adf7d6710ddb8adb6f5c8a00082b117fba6"} Jul 06 12:23:30.125747 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:30.125712 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" event={"ID":"8c8bb9d6-7ec8-495f-a597-4c32a895fd0a","Type":"ContainerStarted","Data":"baa36b82279a05a773a5de6e14f41ca0e8dc9b91ea70e22634d944f5c233ffca"} Jul 06 12:23:30.147064 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:30.147026 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mh96" podStartSLOduration=3.556045286 podStartE2EDuration="25.147013386s" totalImagesPullingTime="21.5909681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:30.146896379 +0000 UTC m=+26.225444407" watchObservedRunningTime="2026-07-06 12:23:30.147013386 +0000 UTC m=+26.225561412" Jul 06 12:23:30.825970 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:30.825940 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:30.826148 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:30.825940 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:30.826148 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:30.826040 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:30.826148 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:30.826124 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:30.890577 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:30.890549 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:30.891111 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:30.891094 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:31.128483 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:31.128416 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:31.128903 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:31.128792 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gkr2l" Jul 06 12:23:31.209500 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:31.209438 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:31.826027 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:31.825855 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:31.826166 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:31.826142 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:32.136645 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:32.136588 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" event={"ID":"64a4bf7d-026e-44d8-bbed-e9c6a2afca09","Type":"ContainerStarted","Data":"be3f86ca0efe1f31674221240f41db917f16aba64d4411319cd31e9c9355fdcf"} Jul 06 12:23:32.184211 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:32.184164 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" podStartSLOduration=7.404133105 podStartE2EDuration="27.184146166s" totalImagesPullingTime="19.780013061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:32.18364939 +0000 UTC m=+28.262197417" watchObservedRunningTime="2026-07-06 12:23:32.184146166 +0000 UTC m=+28.262694228" Jul 06 12:23:32.825591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:32.825555 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:32.825591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:32.825598 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:32.825814 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:32.825705 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:32.825868 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:32.825826 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:33.139723 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:33.139640 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:33.140317 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:33.139738 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:33.140317 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:33.139786 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:33.154590 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:33.154568 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:33.156159 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:33.156139 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:23:33.826004 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:33.825978 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:33.826118 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:33.826084 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:34.143095 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:34.142834 2572 generic.go:356] "Generic (PLEG): container finished" podID="2788ceaa-8992-4990-9bf3-abcb9cc9d57f" containerID="3d4acfbf35b7089cc330d07ba5987427360a26ab0817ac4fab40c7fccf48665f" exitCode=0 Jul 06 12:23:34.143095 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:34.142910 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" event={"ID":"2788ceaa-8992-4990-9bf3-abcb9cc9d57f","Type":"ContainerDied","Data":"3d4acfbf35b7089cc330d07ba5987427360a26ab0817ac4fab40c7fccf48665f"} Jul 06 12:23:34.144346 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:34.144324 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8x6cj"] Jul 06 12:23:34.144445 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:34.144426 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:34.144569 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:34.144507 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:34.146520 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:34.146502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f5mqz"] Jul 06 12:23:34.146594 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:34.146587 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:34.146693 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:34.146655 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:34.161599 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:34.161575 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-scpmd"] Jul 06 12:23:34.161716 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:34.161688 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:34.161796 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:34.161776 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:35.147052 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:35.146969 2572 generic.go:356] "Generic (PLEG): container finished" podID="2788ceaa-8992-4990-9bf3-abcb9cc9d57f" containerID="e7c5623e694c0941d4cb23656590f22eebaefa67daa25e3bf133a71c4815a101" exitCode=0 Jul 06 12:23:35.147052 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:35.147036 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" event={"ID":"2788ceaa-8992-4990-9bf3-abcb9cc9d57f","Type":"ContainerDied","Data":"e7c5623e694c0941d4cb23656590f22eebaefa67daa25e3bf133a71c4815a101"} Jul 06 12:23:35.826250 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:35.826224 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:35.826396 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:35.826312 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:35.826461 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:35.826393 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:35.826511 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:35.826492 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:35.826636 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:35.826612 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:35.826737 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:35.826717 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:36.151921 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:36.151842 2572 generic.go:356] "Generic (PLEG): container finished" podID="2788ceaa-8992-4990-9bf3-abcb9cc9d57f" containerID="42d996bd046885c4efa41e42cbd0eda054abdd89ab67f33307e4cb5522eb8115" exitCode=0 Jul 06 12:23:36.151921 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:36.151882 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" event={"ID":"2788ceaa-8992-4990-9bf3-abcb9cc9d57f","Type":"ContainerDied","Data":"42d996bd046885c4efa41e42cbd0eda054abdd89ab67f33307e4cb5522eb8115"} Jul 06 12:23:37.505634 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:37.505594 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:37.606761 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:37.606730 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:37.825967 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:37.825879 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:37.825967 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:37.825927 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:37.826185 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:37.825887 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:37.826185 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:37.826004 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scpmd" podUID="acffbace-3fd7-4848-8a5b-3b24904b6c1b" Jul 06 12:23:37.826185 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:37.826082 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x6cj" podUID="40839c53-c59a-46b6-8299-4a2ed1c4ec90" Jul 06 12:23:37.826185 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:37.826164 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5mqz" podUID="2598c5d0-e3bb-4604-bd89-064fcf5e0319" Jul 06 12:23:38.558505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:38.558468 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:38.558505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:38.558511 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:38.559066 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:38.558617 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:38.559066 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:38.558622 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:38.559066 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:38.558687 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret podName:40839c53-c59a-46b6-8299-4a2ed1c4ec90 nodeName:}" failed. No retries permitted until 2026-07-06 12:24:10.558668983 +0000 UTC m=+66.637217015 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret") pod "global-pull-secret-syncer-8x6cj" (UID: "40839c53-c59a-46b6-8299-4a2ed1c4ec90") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:23:38.559066 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:38.558709 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs podName:2598c5d0-e3bb-4604-bd89-064fcf5e0319 nodeName:}" failed. No retries permitted until 2026-07-06 12:24:10.558701988 +0000 UTC m=+66.637249993 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs") pod "network-metrics-daemon-f5mqz" (UID: "2598c5d0-e3bb-4604-bd89-064fcf5e0319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:23:38.659404 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:38.659370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:38.659591 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:38.659507 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:23:38.659591 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:38.659534 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:23:38.659591 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:38.659544 2572 projected.go:196] Error preparing data for projected volume kube-api-access-rn55r for pod openshift-network-diagnostics/network-check-target-scpmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:38.659744 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:38.659593 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r podName:acffbace-3fd7-4848-8a5b-3b24904b6c1b nodeName:}" failed. No retries permitted until 2026-07-06 12:24:10.659577817 +0000 UTC m=+66.738125831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rn55r" (UniqueName: "kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r") pod "network-check-target-scpmd" (UID: "acffbace-3fd7-4848-8a5b-3b24904b6c1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:23:39.684258 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.684225 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeReady" Jul 06 12:23:39.684781 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.684370 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 12:23:39.778798 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.778714 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vf2c7"] Jul 06 12:23:39.822398 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.822366 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hmvrc"] Jul 06 12:23:39.822575 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.822494 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vf2c7" Jul 06 12:23:39.826275 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.826250 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 12:23:39.826449 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.826316 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ll78r\"" Jul 06 12:23:39.827071 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.827051 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 12:23:39.827390 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.827372 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 12:23:39.835563 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.835502 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:23:39.835563 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.835528 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:23:39.835563 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.835528 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:39.835824 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.835533 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:23:39.839178 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.839150 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vf2c7"] Jul 06 12:23:39.839291 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.839186 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmvrc"] Jul 06 12:23:39.841526 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.841505 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 12:23:39.841634 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.841605 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ljjwj\"" Jul 06 12:23:39.842097 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.841908 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 12:23:39.842097 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.841931 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 12:23:39.843368 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.843331 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l5tqd\"" Jul 06 12:23:39.843522 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.843381 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 12:23:39.843594 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.843534 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gt55k\"" Jul 06 12:23:39.843682 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.843651 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 12:23:39.844588 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.844572 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 12:23:39.972237 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.972200 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b90c0d-5b63-442c-add0-54a38a3c435e-tmp-dir\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:39.972425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.972254 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef5ec50a-7bdc-4ba5-9218-f15178c889b5-cert\") pod \"ingress-canary-vf2c7\" (UID: \"ef5ec50a-7bdc-4ba5-9218-f15178c889b5\") " pod="openshift-ingress-canary/ingress-canary-vf2c7" Jul 06 12:23:39.972425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.972283 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqmd\" (UniqueName: \"kubernetes.io/projected/ef5ec50a-7bdc-4ba5-9218-f15178c889b5-kube-api-access-sbqmd\") pod \"ingress-canary-vf2c7\" (UID: \"ef5ec50a-7bdc-4ba5-9218-f15178c889b5\") " pod="openshift-ingress-canary/ingress-canary-vf2c7" Jul 06 12:23:39.972425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.972311 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b90c0d-5b63-442c-add0-54a38a3c435e-config-volume\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:39.972425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.972334 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:39.972425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.972360 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnbp5\" (UniqueName: \"kubernetes.io/projected/a8b90c0d-5b63-442c-add0-54a38a3c435e-kube-api-access-bnbp5\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:39.987084 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:39.987057 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:40.073781 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.073696 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b90c0d-5b63-442c-add0-54a38a3c435e-tmp-dir\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:40.073781 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.073760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef5ec50a-7bdc-4ba5-9218-f15178c889b5-cert\") pod \"ingress-canary-vf2c7\" (UID: \"ef5ec50a-7bdc-4ba5-9218-f15178c889b5\") " pod="openshift-ingress-canary/ingress-canary-vf2c7" Jul 06 12:23:40.073998 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.073829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqmd\" (UniqueName: \"kubernetes.io/projected/ef5ec50a-7bdc-4ba5-9218-f15178c889b5-kube-api-access-sbqmd\") pod \"ingress-canary-vf2c7\" (UID: \"ef5ec50a-7bdc-4ba5-9218-f15178c889b5\") " pod="openshift-ingress-canary/ingress-canary-vf2c7" Jul 06 12:23:40.073998 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.073874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b90c0d-5b63-442c-add0-54a38a3c435e-config-volume\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:40.073998 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.073904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:40.073998 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.073936 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnbp5\" (UniqueName: \"kubernetes.io/projected/a8b90c0d-5b63-442c-add0-54a38a3c435e-kube-api-access-bnbp5\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:40.074136 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:40.074085 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:23:40.074168 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:40.074141 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls podName:a8b90c0d-5b63-442c-add0-54a38a3c435e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:40.574122071 +0000 UTC m=+36.652670076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls") pod "dns-default-hmvrc" (UID: "a8b90c0d-5b63-442c-add0-54a38a3c435e") : secret "dns-default-metrics-tls" not found Jul 06 12:23:40.074414 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.074394 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b90c0d-5b63-442c-add0-54a38a3c435e-config-volume\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:40.074549 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.074521 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b90c0d-5b63-442c-add0-54a38a3c435e-tmp-dir\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:40.079067 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.079028 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef5ec50a-7bdc-4ba5-9218-f15178c889b5-cert\") pod \"ingress-canary-vf2c7\" (UID: \"ef5ec50a-7bdc-4ba5-9218-f15178c889b5\") " pod="openshift-ingress-canary/ingress-canary-vf2c7" Jul 06 12:23:40.091501 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.091481 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:40.091612 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.091567 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqmd\" (UniqueName: \"kubernetes.io/projected/ef5ec50a-7bdc-4ba5-9218-f15178c889b5-kube-api-access-sbqmd\") pod \"ingress-canary-vf2c7\" (UID: \"ef5ec50a-7bdc-4ba5-9218-f15178c889b5\") " pod="openshift-ingress-canary/ingress-canary-vf2c7" Jul 06 12:23:40.092176 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.092129 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnbp5\" (UniqueName: \"kubernetes.io/projected/a8b90c0d-5b63-442c-add0-54a38a3c435e-kube-api-access-bnbp5\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:40.132958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.132931 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vf2c7" Jul 06 12:23:40.578954 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.578917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:40.579115 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:40.579062 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:23:40.579154 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:40.579134 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls podName:a8b90c0d-5b63-442c-add0-54a38a3c435e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:41.579117883 +0000 UTC m=+37.657665888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls") pod "dns-default-hmvrc" (UID: "a8b90c0d-5b63-442c-add0-54a38a3c435e") : secret "dns-default-metrics-tls" not found Jul 06 12:23:40.813355 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:40.813162 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vf2c7"] Jul 06 12:23:40.895282 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:40.895222 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5ec50a_7bdc_4ba5_9218_f15178c889b5.slice/crio-a7b94686d39df036149f5566f686936e13e18d18acfd50858385943c04be655e WatchSource:0}: Error finding container a7b94686d39df036149f5566f686936e13e18d18acfd50858385943c04be655e: Status 404 returned error can't find the container with id a7b94686d39df036149f5566f686936e13e18d18acfd50858385943c04be655e Jul 06 12:23:41.165904 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:41.165839 2572 generic.go:356] "Generic (PLEG): container finished" podID="2788ceaa-8992-4990-9bf3-abcb9cc9d57f" containerID="0df1d3c7ef02cb1febcbe24d33147057ce94a8ed96880cd8f11f56af2c02fc03" exitCode=0 Jul 06 12:23:41.166003 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:41.165924 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" event={"ID":"2788ceaa-8992-4990-9bf3-abcb9cc9d57f","Type":"ContainerDied","Data":"0df1d3c7ef02cb1febcbe24d33147057ce94a8ed96880cd8f11f56af2c02fc03"} Jul 06 12:23:41.167084 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:41.167056 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vf2c7" event={"ID":"ef5ec50a-7bdc-4ba5-9218-f15178c889b5","Type":"ContainerStarted","Data":"a7b94686d39df036149f5566f686936e13e18d18acfd50858385943c04be655e"} Jul 06 12:23:41.585879 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:41.585842 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:41.586062 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:41.585996 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:23:41.586135 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:41.586073 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls podName:a8b90c0d-5b63-442c-add0-54a38a3c435e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:43.586050857 +0000 UTC m=+39.664598867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls") pod "dns-default-hmvrc" (UID: "a8b90c0d-5b63-442c-add0-54a38a3c435e") : secret "dns-default-metrics-tls" not found Jul 06 12:23:42.173241 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:42.173205 2572 generic.go:356] "Generic (PLEG): container finished" podID="2788ceaa-8992-4990-9bf3-abcb9cc9d57f" containerID="bdd6470d83f151b0b9f6d24ed0fca58cd69f9207c63d221ebdc6023776011f36" exitCode=0 Jul 06 12:23:42.173739 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:42.173260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" event={"ID":"2788ceaa-8992-4990-9bf3-abcb9cc9d57f","Type":"ContainerDied","Data":"bdd6470d83f151b0b9f6d24ed0fca58cd69f9207c63d221ebdc6023776011f36"} Jul 06 12:23:43.179311 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:43.179260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" event={"ID":"2788ceaa-8992-4990-9bf3-abcb9cc9d57f","Type":"ContainerStarted","Data":"37889c9ef3ce286b3ddf6b8239786b70ab6e27decf07d8eb9ebca158c4e8ee7e"} Jul 06 12:23:43.216708 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:43.216638 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rbfcf" podStartSLOduration=5.935144641 podStartE2EDuration="37.216618804s" totalImagesPullingTime="31.281474163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:43.214537501 +0000 UTC m=+39.293085530" watchObservedRunningTime="2026-07-06 12:23:43.216618804 +0000 UTC m=+39.295166831" Jul 06 12:23:43.359183 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:43.359152 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:43.444699 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:43.444613 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:43.545034 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:43.545005 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:43.602143 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:43.602116 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:43.602259 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:43.602215 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:23:43.602296 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:43.602264 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls podName:a8b90c0d-5b63-442c-add0-54a38a3c435e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:47.602251278 +0000 UTC m=+43.680799283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls") pod "dns-default-hmvrc" (UID: "a8b90c0d-5b63-442c-add0-54a38a3c435e") : secret "dns-default-metrics-tls" not found Jul 06 12:23:44.183649 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:44.183364 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vf2c7" event={"ID":"ef5ec50a-7bdc-4ba5-9218-f15178c889b5","Type":"ContainerStarted","Data":"1f7b9b91094b5dad401f0b9507b489b8f6faa14ca1cff256f2283968d763ab67"} Jul 06 12:23:44.200728 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:44.200685 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vf2c7" podStartSLOduration=2.375894872 podStartE2EDuration="5.200672024s" totalImagesPullingTime="2.824777152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:44.200102477 +0000 UTC m=+40.278650503" watchObservedRunningTime="2026-07-06 12:23:44.200672024 +0000 UTC m=+40.279220043" Jul 06 12:23:44.618042 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:44.618013 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:44.685156 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:44.685129 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:47.629959 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:47.629922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:47.630340 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:47.630070 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:23:47.630340 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:47.630129 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls podName:a8b90c0d-5b63-442c-add0-54a38a3c435e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:55.630115068 +0000 UTC m=+51.708663076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls") pod "dns-default-hmvrc" (UID: "a8b90c0d-5b63-442c-add0-54a38a3c435e") : secret "dns-default-metrics-tls" not found Jul 06 12:23:48.179566 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.179534 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x"] Jul 06 12:23:48.211425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.211396 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x"] Jul 06 12:23:48.211572 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.211481 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.216387 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.216363 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 12:23:48.216498 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.216407 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 12:23:48.216741 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.216725 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 12:23:48.216835 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.216821 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 12:23:48.217562 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.217542 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xkkck\"" Jul 06 12:23:48.276621 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.276587 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9c745db8b-dhqbq"] Jul 06 12:23:48.297376 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.297349 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc"] Jul 06 12:23:48.297495 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.297459 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.301591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.301567 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 12:23:48.301971 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.301954 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mnq4l\"" Jul 06 12:23:48.302116 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.302100 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 12:23:48.302191 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.302124 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 12:23:48.322974 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.322938 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9c745db8b-dhqbq"] Jul 06 12:23:48.323096 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.323054 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:48.324250 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.324222 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 12:23:48.327110 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.327090 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 12:23:48.327209 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.327192 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jnrhh\"" Jul 06 12:23:48.327265 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.327193 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 12:23:48.327990 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.327973 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 12:23:48.327990 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.327982 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 12:23:48.335408 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.335386 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfc2h\" (UniqueName: \"kubernetes.io/projected/9b696dd3-c3e9-47c5-b1fe-366497f5b1e3-kube-api-access-hfc2h\") pod \"kube-storage-version-migrator-operator-846bf668f7-qdm6x\" (UID: \"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.335545 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.335423 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b696dd3-c3e9-47c5-b1fe-366497f5b1e3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-qdm6x\" (UID: \"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.335545 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.335459 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b696dd3-c3e9-47c5-b1fe-366497f5b1e3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-qdm6x\" (UID: \"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.336373 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.336353 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc"] Jul 06 12:23:48.436039 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.435968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.436039 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436000 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-registry-certificates\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.436039 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436023 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2f6\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-kube-api-access-wv2f6\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.436221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436041 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68dwz\" (UniqueName: \"kubernetes.io/projected/f0caf416-8c07-442a-8c58-f06069227b50-kube-api-access-68dwz\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:48.436221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436111 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:48.436221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfc2h\" (UniqueName: \"kubernetes.io/projected/9b696dd3-c3e9-47c5-b1fe-366497f5b1e3-kube-api-access-hfc2h\") pod \"kube-storage-version-migrator-operator-846bf668f7-qdm6x\" (UID: \"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.436221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-bound-sa-token\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.436221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b696dd3-c3e9-47c5-b1fe-366497f5b1e3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-qdm6x\" (UID: \"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.436221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436204 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b696dd3-c3e9-47c5-b1fe-366497f5b1e3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-qdm6x\" (UID: \"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.436221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436222 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-image-registry-private-configuration\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.436418 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436340 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-installation-pull-secrets\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.436418 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436370 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-trusted-ca\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.436418 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436386 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0caf416-8c07-442a-8c58-f06069227b50-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:48.436524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436415 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a06ef38-4591-4855-837e-f012535c3af4-ca-trust-extracted\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.436838 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.436816 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b696dd3-c3e9-47c5-b1fe-366497f5b1e3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-qdm6x\" (UID: \"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.439857 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.439828 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b696dd3-c3e9-47c5-b1fe-366497f5b1e3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-qdm6x\" (UID: \"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.449065 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.449042 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfc2h\" (UniqueName: \"kubernetes.io/projected/9b696dd3-c3e9-47c5-b1fe-366497f5b1e3-kube-api-access-hfc2h\") pod \"kube-storage-version-migrator-operator-846bf668f7-qdm6x\" (UID: \"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.521047 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.521021 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" Jul 06 12:23:48.536742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.536715 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-bound-sa-token\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.536841 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.536749 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-image-registry-private-configuration\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.536841 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.536778 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-installation-pull-secrets\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.536841 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.536803 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-trusted-ca\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.536841 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.536820 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0caf416-8c07-442a-8c58-f06069227b50-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:48.537041 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.536843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a06ef38-4591-4855-837e-f012535c3af4-ca-trust-extracted\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.537097 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.537066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.537152 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.537102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-registry-certificates\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.537152 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.537127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2f6\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-kube-api-access-wv2f6\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.537152 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.537143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68dwz\" (UniqueName: \"kubernetes.io/projected/f0caf416-8c07-442a-8c58-f06069227b50-kube-api-access-68dwz\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:48.537297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.537191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:48.537297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.537266 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a06ef38-4591-4855-837e-f012535c3af4-ca-trust-extracted\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.537297 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:48.537283 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:48.537442 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:48.537349 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls podName:f0caf416-8c07-442a-8c58-f06069227b50 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:49.037330785 +0000 UTC m=+45.115878801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-sgkbc" (UID: "f0caf416-8c07-442a-8c58-f06069227b50") : secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:48.537442 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:48.537365 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:23:48.537442 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:48.537379 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c745db8b-dhqbq: secret "image-registry-tls" not found Jul 06 12:23:48.537442 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:48.537425 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls podName:4a06ef38-4591-4855-837e-f012535c3af4 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:49.037413445 +0000 UTC m=+45.115961454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls") pod "image-registry-9c745db8b-dhqbq" (UID: "4a06ef38-4591-4855-837e-f012535c3af4") : secret "image-registry-tls" not found Jul 06 12:23:48.537711 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.537680 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0caf416-8c07-442a-8c58-f06069227b50-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:48.537811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.537784 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-registry-certificates\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.538076 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.538052 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:48.538350 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.538328 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-trusted-ca\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.539438 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.539419 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-image-registry-private-configuration\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.539726 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.539711 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-installation-pull-secrets\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.556075 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.556053 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-bound-sa-token\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.557091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.557071 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68dwz\" (UniqueName: \"kubernetes.io/projected/f0caf416-8c07-442a-8c58-f06069227b50-kube-api-access-68dwz\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:48.558802 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.558786 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2f6\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-kube-api-access-wv2f6\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:48.648229 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.648202 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:48.670631 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:48.670606 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x"] Jul 06 12:23:48.673527 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:48.673502 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b696dd3_c3e9_47c5_b1fe_366497f5b1e3.slice/crio-01c6279bd2486d3826be6e6c373a1dbd58161941da1a8e286f494c48af38c496 WatchSource:0}: Error finding container 01c6279bd2486d3826be6e6c373a1dbd58161941da1a8e286f494c48af38c496: Status 404 returned error can't find the container with id 01c6279bd2486d3826be6e6c373a1dbd58161941da1a8e286f494c48af38c496 Jul 06 12:23:49.041758 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:49.041728 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:49.041921 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:49.041800 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:49.041921 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:49.041884 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:49.041921 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:49.041904 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:23:49.041921 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:49.041914 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c745db8b-dhqbq: secret "image-registry-tls" not found Jul 06 12:23:49.042045 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:49.041954 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls podName:f0caf416-8c07-442a-8c58-f06069227b50 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:50.041933646 +0000 UTC m=+46.120481679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-sgkbc" (UID: "f0caf416-8c07-442a-8c58-f06069227b50") : secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:49.042045 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:49.041972 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls podName:4a06ef38-4591-4855-837e-f012535c3af4 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:50.041965185 +0000 UTC m=+46.120513190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls") pod "image-registry-9c745db8b-dhqbq" (UID: "4a06ef38-4591-4855-837e-f012535c3af4") : secret "image-registry-tls" not found Jul 06 12:23:49.194068 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:49.194034 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" event={"ID":"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3","Type":"ContainerStarted","Data":"01c6279bd2486d3826be6e6c373a1dbd58161941da1a8e286f494c48af38c496"} Jul 06 12:23:49.405820 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:49.405732 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:49.517367 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:49.517332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:49.785586 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:49.785547 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:49.882779 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:49.882750 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:50.048711 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:50.048638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:50.048835 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:50.048761 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:50.048835 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:50.048793 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:50.048909 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:50.048866 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls podName:f0caf416-8c07-442a-8c58-f06069227b50 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:52.048847402 +0000 UTC m=+48.127395407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-sgkbc" (UID: "f0caf416-8c07-442a-8c58-f06069227b50") : secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:50.048909 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:50.048865 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:23:50.048909 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:50.048882 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c745db8b-dhqbq: secret "image-registry-tls" not found Jul 06 12:23:50.048909 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:50.048906 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls podName:4a06ef38-4591-4855-837e-f012535c3af4 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:52.04889956 +0000 UTC m=+48.127447565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls") pod "image-registry-9c745db8b-dhqbq" (UID: "4a06ef38-4591-4855-837e-f012535c3af4") : secret "image-registry-tls" not found Jul 06 12:23:52.062059 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.062020 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:52.062520 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.062094 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:52.062520 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:52.062159 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:52.062520 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:52.062180 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:23:52.062520 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:52.062191 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c745db8b-dhqbq: secret "image-registry-tls" not found Jul 06 12:23:52.062520 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:52.062220 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls podName:f0caf416-8c07-442a-8c58-f06069227b50 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:56.062203417 +0000 UTC m=+52.140751423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-sgkbc" (UID: "f0caf416-8c07-442a-8c58-f06069227b50") : secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:52.062520 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:52.062234 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls podName:4a06ef38-4591-4855-837e-f012535c3af4 nodeName:}" failed. No retries permitted until 2026-07-06 12:23:56.062227796 +0000 UTC m=+52.140775801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls") pod "image-registry-9c745db8b-dhqbq" (UID: "4a06ef38-4591-4855-837e-f012535c3af4") : secret "image-registry-tls" not found Jul 06 12:23:52.206203 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.206166 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" event={"ID":"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3","Type":"ContainerStarted","Data":"3ca3487f3916989730eca89b87feef44f609a9b2f071027d762d50b878be1285"} Jul 06 12:23:52.227989 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.227942 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" podStartSLOduration=1.465896696 podStartE2EDuration="4.227926571s" totalImagesPullingTime="2.762029875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:52.226720603 +0000 UTC m=+48.305268634" watchObservedRunningTime="2026-07-06 12:23:52.227926571 +0000 UTC m=+48.306474598" Jul 06 12:23:52.274862 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.274827 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:52.378812 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.378746 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:52.470186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.470154 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lrf5l"] Jul 06 12:23:52.505239 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.505214 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lrf5l"] Jul 06 12:23:52.505379 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.505340 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.505573 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.505533 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:52.509783 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.509763 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 12:23:52.510137 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.510119 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 12:23:52.510587 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.510563 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9zwn2\"" Jul 06 12:23:52.512019 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.511896 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 12:23:52.512289 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.512271 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 12:23:52.566416 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.566380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.566570 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.566423 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9l4\" (UniqueName: \"kubernetes.io/projected/c970c174-8210-443d-a38c-7542aeac0e1e-kube-api-access-wd9l4\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.566570 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.566472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c970c174-8210-443d-a38c-7542aeac0e1e-data-volume\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.566570 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.566488 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.566570 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.566533 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c970c174-8210-443d-a38c-7542aeac0e1e-crio-socket\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.591727 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.591699 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:52.667745 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.667631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c970c174-8210-443d-a38c-7542aeac0e1e-data-volume\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.667745 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.667690 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.667745 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.667731 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c970c174-8210-443d-a38c-7542aeac0e1e-crio-socket\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.668011 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.667771 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.668011 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.667810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9l4\" (UniqueName: \"kubernetes.io/projected/c970c174-8210-443d-a38c-7542aeac0e1e-kube-api-access-wd9l4\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.668011 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:52.667849 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:23:52.668011 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:52.667871 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:23:52.668011 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:52.667918 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:53.167897904 +0000 UTC m=+49.246445914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : secret "insights-runtime-extractor-tls" not found Jul 06 12:23:52.668011 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:52.667937 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:53.167927495 +0000 UTC m=+49.246475503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : configmap "kube-rbac-proxy" not found Jul 06 12:23:52.668011 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.667955 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c970c174-8210-443d-a38c-7542aeac0e1e-crio-socket\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.668011 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.667991 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c970c174-8210-443d-a38c-7542aeac0e1e-data-volume\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.678187 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.678154 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9l4\" (UniqueName: \"kubernetes.io/projected/c970c174-8210-443d-a38c-7542aeac0e1e-kube-api-access-wd9l4\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:52.938017 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.937945 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:52.979228 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.979203 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb"] Jul 06 12:23:52.999797 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.999773 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb"] Jul 06 12:23:52.999940 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:52.999870 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb" Jul 06 12:23:53.004172 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.004146 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xfn8w\"" Jul 06 12:23:53.004312 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.004176 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 12:23:53.004853 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.004832 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 12:23:53.025617 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.025593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:53.071366 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.071341 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rklnc\" (UniqueName: \"kubernetes.io/projected/0268860c-bc12-4a55-aa9c-10e02d20b683-kube-api-access-rklnc\") pod \"migrator-d57d58cc9-98pbb\" (UID: \"0268860c-bc12-4a55-aa9c-10e02d20b683\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb" Jul 06 12:23:53.112219 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.112192 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:53.171641 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.171615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rklnc\" (UniqueName: \"kubernetes.io/projected/0268860c-bc12-4a55-aa9c-10e02d20b683-kube-api-access-rklnc\") pod \"migrator-d57d58cc9-98pbb\" (UID: \"0268860c-bc12-4a55-aa9c-10e02d20b683\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb" Jul 06 12:23:53.171790 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.171699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:53.171849 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.171803 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:53.171849 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:53.171826 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:23:53.171929 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:53.171906 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:54.171885402 +0000 UTC m=+50.250433411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : secret "insights-runtime-extractor-tls" not found Jul 06 12:23:53.171929 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:53.171917 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:23:53.172012 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:53.171983 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:54.17196984 +0000 UTC m=+50.250517845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : configmap "kube-rbac-proxy" not found Jul 06 12:23:53.181390 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.181362 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rklnc\" (UniqueName: \"kubernetes.io/projected/0268860c-bc12-4a55-aa9c-10e02d20b683-kube-api-access-rklnc\") pod \"migrator-d57d58cc9-98pbb\" (UID: \"0268860c-bc12-4a55-aa9c-10e02d20b683\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb" Jul 06 12:23:53.242428 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.242399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:53.308261 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.308235 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb" Jul 06 12:23:53.338458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.338437 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:53.447418 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:53.447380 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb"] Jul 06 12:23:53.450003 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:23:53.449972 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0268860c_bc12_4a55_aa9c_10e02d20b683.slice/crio-ac7fad1162ae2ec24307f920e3888fd7073e6824a7a161f4b7d8c9a6b13f16fc WatchSource:0}: Error finding container ac7fad1162ae2ec24307f920e3888fd7073e6824a7a161f4b7d8c9a6b13f16fc: Status 404 returned error can't find the container with id ac7fad1162ae2ec24307f920e3888fd7073e6824a7a161f4b7d8c9a6b13f16fc Jul 06 12:23:54.180110 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:54.180081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:54.180584 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:54.180244 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:54.180584 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:54.180259 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:23:54.180584 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:54.180334 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:56.180313741 +0000 UTC m=+52.258861748 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : secret "insights-runtime-extractor-tls" not found Jul 06 12:23:54.180584 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:54.180363 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:23:54.180584 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:54.180412 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:23:56.180396601 +0000 UTC m=+52.258944609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : configmap "kube-rbac-proxy" not found Jul 06 12:23:54.212142 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:54.212108 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb" event={"ID":"0268860c-bc12-4a55-aa9c-10e02d20b683","Type":"ContainerStarted","Data":"ac7fad1162ae2ec24307f920e3888fd7073e6824a7a161f4b7d8c9a6b13f16fc"} Jul 06 12:23:54.232982 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:54.232958 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fbxl6_fddefa28-3b87-4803-833e-818c0d54b665/dns-node-resolver/0.log" Jul 06 12:23:54.868291 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:54.868264 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:54.948304 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:54.948281 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:55.215751 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:55.215723 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb" event={"ID":"0268860c-bc12-4a55-aa9c-10e02d20b683","Type":"ContainerStarted","Data":"74e7da3eb304514330a45b6ff7957207d6a7cb8bb86ab2ff1715a94728ed9081"} Jul 06 12:23:55.216092 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:55.215769 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb" event={"ID":"0268860c-bc12-4a55-aa9c-10e02d20b683","Type":"ContainerStarted","Data":"9b91c6434a1e2f0d9c256c953c6a8a8567913e0a6b329576fcf7f32ffbb16333"} Jul 06 12:23:55.234352 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:55.234329 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xqvf_5302ffb2-87cb-4aba-8e8a-0e7208e8b217/node-ca/0.log" Jul 06 12:23:55.234635 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:55.234601 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-98pbb" podStartSLOduration=1.87678325 podStartE2EDuration="3.234586984s" totalImagesPullingTime="1.357803734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:23:55.232222106 +0000 UTC m=+51.310770136" watchObservedRunningTime="2026-07-06 12:23:55.234586984 +0000 UTC m=+51.313135012" Jul 06 12:23:55.330032 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:55.330007 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:55.691737 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:55.691647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:23:55.691894 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:55.691802 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:23:55.691894 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:55.691862 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls podName:a8b90c0d-5b63-442c-add0-54a38a3c435e nodeName:}" failed. No retries permitted until 2026-07-06 12:24:11.691846686 +0000 UTC m=+67.770394692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls") pod "dns-default-hmvrc" (UID: "a8b90c0d-5b63-442c-add0-54a38a3c435e") : secret "dns-default-metrics-tls" not found Jul 06 12:23:55.836966 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:55.836933 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:23:56.094890 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:56.094856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:23:56.095076 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:56.094942 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:23:56.095076 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:56.095028 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:56.095204 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:56.095107 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls podName:f0caf416-8c07-442a-8c58-f06069227b50 nodeName:}" failed. No retries permitted until 2026-07-06 12:24:04.095085942 +0000 UTC m=+60.173633960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-sgkbc" (UID: "f0caf416-8c07-442a-8c58-f06069227b50") : secret "cluster-monitoring-operator-tls" not found Jul 06 12:23:56.095204 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:56.095036 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:23:56.095204 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:56.095132 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c745db8b-dhqbq: secret "image-registry-tls" not found Jul 06 12:23:56.095204 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:56.095185 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls podName:4a06ef38-4591-4855-837e-f012535c3af4 nodeName:}" failed. No retries permitted until 2026-07-06 12:24:04.095170193 +0000 UTC m=+60.173718199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls") pod "image-registry-9c745db8b-dhqbq" (UID: "4a06ef38-4591-4855-837e-f012535c3af4") : secret "image-registry-tls" not found Jul 06 12:23:56.196174 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:56.196129 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:56.196340 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:56.196254 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:23:56.196340 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:56.196311 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:23:56.196450 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:56.196391 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:24:00.196371807 +0000 UTC m=+56.274919812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : configmap "kube-rbac-proxy" not found Jul 06 12:23:56.196450 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:56.196405 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:23:56.196521 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:23:56.196456 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:24:00.196444627 +0000 UTC m=+56.274992636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : secret "insights-runtime-extractor-tls" not found Jul 06 12:23:56.433197 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:56.433124 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vf2c7_ef5ec50a-7bdc-4ba5-9218-f15178c889b5/serve-healthcheck-canary/0.log" Jul 06 12:23:58.742783 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:23:58.742749 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:00.227540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:00.227503 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:24:00.227980 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:00.227578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:24:00.227980 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:00.227695 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:24:00.227980 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:00.227712 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:24:00.227980 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:00.227770 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:24:08.227749765 +0000 UTC m=+64.306297773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : secret "insights-runtime-extractor-tls" not found Jul 06 12:24:00.227980 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:00.227791 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:24:08.227782091 +0000 UTC m=+64.306330096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : configmap "kube-rbac-proxy" not found Jul 06 12:24:02.617479 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:02.617448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:04.158223 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:04.158187 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:24:04.158610 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:04.158343 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 12:24:04.158610 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:04.158345 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls\") pod \"image-registry-9c745db8b-dhqbq\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:24:04.158610 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:04.158417 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls podName:f0caf416-8c07-442a-8c58-f06069227b50 nodeName:}" failed. No retries permitted until 2026-07-06 12:24:20.15839587 +0000 UTC m=+76.236943874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-sgkbc" (UID: "f0caf416-8c07-442a-8c58-f06069227b50") : secret "cluster-monitoring-operator-tls" not found Jul 06 12:24:04.158610 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:04.158419 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:24:04.158610 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:04.158462 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c745db8b-dhqbq: secret "image-registry-tls" not found Jul 06 12:24:04.158610 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:04.158518 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls podName:4a06ef38-4591-4855-837e-f012535c3af4 nodeName:}" failed. No retries permitted until 2026-07-06 12:24:20.158503364 +0000 UTC m=+76.237051369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls") pod "image-registry-9c745db8b-dhqbq" (UID: "4a06ef38-4591-4855-837e-f012535c3af4") : secret "image-registry-tls" not found Jul 06 12:24:05.165202 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:05.165178 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnmxh" Jul 06 12:24:05.681865 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:05.681834 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:08.291791 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:08.291755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:24:08.292167 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:08.291903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:24:08.292167 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:08.292001 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:24:08.292167 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:08.292052 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls podName:c970c174-8210-443d-a38c-7542aeac0e1e nodeName:}" failed. No retries permitted until 2026-07-06 12:24:24.292036182 +0000 UTC m=+80.370584189 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrf5l" (UID: "c970c174-8210-443d-a38c-7542aeac0e1e") : secret "insights-runtime-extractor-tls" not found Jul 06 12:24:08.292284 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:08.292263 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c970c174-8210-443d-a38c-7542aeac0e1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:24:09.405869 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:09.405833 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:09.602842 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:09.602810 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:09.703835 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:09.703760 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:09.855015 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:09.854986 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:09.985134 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:09.985103 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:10.119819 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.119791 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:10.244882 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.244819 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:10.608809 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.608729 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:24:10.608809 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.608765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:24:10.613710 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.613689 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 12:24:10.613778 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.613724 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 12:24:10.619771 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:10.619755 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 12:24:10.619863 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:10.619810 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs podName:2598c5d0-e3bb-4604-bd89-064fcf5e0319 nodeName:}" failed. No retries permitted until 2026-07-06 12:25:14.619795236 +0000 UTC m=+130.698343241 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs") pod "network-metrics-daemon-f5mqz" (UID: "2598c5d0-e3bb-4604-bd89-064fcf5e0319") : secret "metrics-daemon-secret" not found Jul 06 12:24:10.621876 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.621853 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40839c53-c59a-46b6-8299-4a2ed1c4ec90-original-pull-secret\") pod \"global-pull-secret-syncer-8x6cj\" (UID: \"40839c53-c59a-46b6-8299-4a2ed1c4ec90\") " pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:24:10.710012 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.709974 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:24:10.713641 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.713623 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 12:24:10.724126 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.724107 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 12:24:10.732907 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.732882 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn55r\" (UniqueName: \"kubernetes.io/projected/acffbace-3fd7-4848-8a5b-3b24904b6c1b-kube-api-access-rn55r\") pod \"network-check-target-scpmd\" (UID: \"acffbace-3fd7-4848-8a5b-3b24904b6c1b\") " pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:24:10.749833 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.749811 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ljjwj\"" Jul 06 12:24:10.757006 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.756990 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:24:10.766145 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.766118 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x6cj" Jul 06 12:24:10.890274 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.890195 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-scpmd"] Jul 06 12:24:10.893247 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:10.893218 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacffbace_3fd7_4848_8a5b_3b24904b6c1b.slice/crio-48d5f95df42628a40a23612b3bdd003731b8b81c3f1f11a74bebe40baadc833c WatchSource:0}: Error finding container 48d5f95df42628a40a23612b3bdd003731b8b81c3f1f11a74bebe40baadc833c: Status 404 returned error can't find the container with id 48d5f95df42628a40a23612b3bdd003731b8b81c3f1f11a74bebe40baadc833c Jul 06 12:24:10.902935 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:10.902914 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8x6cj"] Jul 06 12:24:10.904902 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:10.904881 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40839c53_c59a_46b6_8299_4a2ed1c4ec90.slice/crio-400669fa0668e132847c5c494bd8b4fc51b03b2a20ddd5b537da60c1892334d4 WatchSource:0}: Error finding container 400669fa0668e132847c5c494bd8b4fc51b03b2a20ddd5b537da60c1892334d4: Status 404 returned error can't find the container with id 400669fa0668e132847c5c494bd8b4fc51b03b2a20ddd5b537da60c1892334d4 Jul 06 12:24:11.102795 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:11.102764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:11.259058 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:11.259028 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8x6cj" event={"ID":"40839c53-c59a-46b6-8299-4a2ed1c4ec90","Type":"ContainerStarted","Data":"400669fa0668e132847c5c494bd8b4fc51b03b2a20ddd5b537da60c1892334d4"} Jul 06 12:24:11.260076 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:11.260049 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-scpmd" event={"ID":"acffbace-3fd7-4848-8a5b-3b24904b6c1b","Type":"ContainerStarted","Data":"48d5f95df42628a40a23612b3bdd003731b8b81c3f1f11a74bebe40baadc833c"} Jul 06 12:24:11.547415 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:11.547327 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:11.716835 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:11.716800 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:24:11.717288 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:11.716964 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:24:11.717288 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:11.717016 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls podName:a8b90c0d-5b63-442c-add0-54a38a3c435e nodeName:}" failed. No retries permitted until 2026-07-06 12:24:43.717003599 +0000 UTC m=+99.795551603 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls") pod "dns-default-hmvrc" (UID: "a8b90c0d-5b63-442c-add0-54a38a3c435e") : secret "dns-default-metrics-tls" not found Jul 06 12:24:11.717869 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:11.717830 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:11.912331 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:11.912209 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:12.096934 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:12.096802 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:13.220649 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:13.220560 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:13.400440 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:13.400406 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:13.703582 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:13.703493 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:13.919330 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:13.919293 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:14.020228 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:14.020198 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:14.498316 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:14.498278 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:14.898053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:14.897961 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:15.129630 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:15.129599 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:15.548794 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:15.548766 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:15.763969 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:15.763940 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:16.275888 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.275855 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8x6cj" event={"ID":"40839c53-c59a-46b6-8299-4a2ed1c4ec90","Type":"ContainerStarted","Data":"b4874f5a684f8aa7610b69babccc688c9725e0d67b172c9397fc8fd6a0a45c7a"} Jul 06 12:24:16.277322 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.277285 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-scpmd" event={"ID":"acffbace-3fd7-4848-8a5b-3b24904b6c1b","Type":"ContainerStarted","Data":"9fad77d73e0a8ef3671f681d88d45ec66e3def9211d8d5c125ac61f63dcfc88f"} Jul 06 12:24:16.277461 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.277339 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:24:16.295849 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.295810 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8x6cj" podStartSLOduration=65.76737292 podStartE2EDuration="1m10.295797425s" totalImagesPullingTime="4.528424505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:16.29473955 +0000 UTC m=+72.373287576" watchObservedRunningTime="2026-07-06 12:24:16.295797425 +0000 UTC m=+72.374345842" Jul 06 12:24:16.304304 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.304280 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:16.318068 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.318016 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-scpmd" podStartSLOduration=65.781046477 podStartE2EDuration="1m10.318002139s" totalImagesPullingTime="4.536955662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:16.317918221 +0000 UTC m=+72.396466251" watchObservedRunningTime="2026-07-06 12:24:16.318002139 +0000 UTC m=+72.396550166" Jul 06 12:24:16.772972 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.772916 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:16.816459 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.816434 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp"] Jul 06 12:24:16.818439 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.818425 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.821369 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.821350 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 12:24:16.821628 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.821607 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 12:24:16.822753 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.822736 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 12:24:16.823099 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.823078 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 12:24:16.823201 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.823103 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 12:24:16.823201 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.823103 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 12:24:16.823294 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.823231 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 12:24:16.844805 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.844782 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp"] Jul 06 12:24:16.861390 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.861368 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.861499 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.861398 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-ca\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.861499 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.861414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-hub\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.861499 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.861434 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.861499 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.861486 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btb67\" (UniqueName: \"kubernetes.io/projected/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-kube-api-access-btb67\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.861652 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.861609 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.872917 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.872892 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9c745db8b-dhqbq"] Jul 06 12:24:16.873090 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:16.873072 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" podUID="4a06ef38-4591-4855-837e-f012535c3af4" Jul 06 12:24:16.935036 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.935009 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:16.962770 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.962742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.962946 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.962926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.962997 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.962959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-ca\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.962997 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.962977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-hub\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.963096 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.963009 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.963096 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.963035 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btb67\" (UniqueName: \"kubernetes.io/projected/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-kube-api-access-btb67\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.963640 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.963618 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.965416 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.965391 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.965544 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.965525 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.965738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.965722 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-ca\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.965861 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.965844 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-hub\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:16.989782 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:16.989754 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btb67\" (UniqueName: \"kubernetes.io/projected/22ca1f39-285a-4d4b-953f-ae368fe5d4a8-kube-api-access-btb67\") pod \"cluster-proxy-proxy-agent-84c575594-h8jbp\" (UID: \"22ca1f39-285a-4d4b-953f-ae368fe5d4a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:17.135166 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.135100 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" Jul 06 12:24:17.281031 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.280954 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:24:17.282239 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.282213 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp"] Jul 06 12:24:17.282616 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.282593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:17.285380 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:17.285358 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22ca1f39_285a_4d4b_953f_ae368fe5d4a8.slice/crio-a50f878a4ee7b83fcbcf821d62f6e49be64b4cd595070fb4bf21acd86ca83ea5 WatchSource:0}: Error finding container a50f878a4ee7b83fcbcf821d62f6e49be64b4cd595070fb4bf21acd86ca83ea5: Status 404 returned error can't find the container with id a50f878a4ee7b83fcbcf821d62f6e49be64b4cd595070fb4bf21acd86ca83ea5 Jul 06 12:24:17.287725 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.287709 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:24:17.365805 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.365772 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-registry-certificates\") pod \"4a06ef38-4591-4855-837e-f012535c3af4\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " Jul 06 12:24:17.365922 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.365816 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a06ef38-4591-4855-837e-f012535c3af4-ca-trust-extracted\") pod \"4a06ef38-4591-4855-837e-f012535c3af4\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " Jul 06 12:24:17.365922 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.365853 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv2f6\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-kube-api-access-wv2f6\") pod \"4a06ef38-4591-4855-837e-f012535c3af4\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " Jul 06 12:24:17.365922 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.365893 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-trusted-ca\") pod \"4a06ef38-4591-4855-837e-f012535c3af4\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " Jul 06 12:24:17.366041 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.365928 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-installation-pull-secrets\") pod \"4a06ef38-4591-4855-837e-f012535c3af4\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " Jul 06 12:24:17.366041 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.365976 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-bound-sa-token\") pod \"4a06ef38-4591-4855-837e-f012535c3af4\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " Jul 06 12:24:17.366144 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.366118 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a06ef38-4591-4855-837e-f012535c3af4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4a06ef38-4591-4855-837e-f012535c3af4" (UID: "4a06ef38-4591-4855-837e-f012535c3af4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:24:17.366214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.366149 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-image-registry-private-configuration\") pod \"4a06ef38-4591-4855-837e-f012535c3af4\" (UID: \"4a06ef38-4591-4855-837e-f012535c3af4\") " Jul 06 12:24:17.366267 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.366156 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4a06ef38-4591-4855-837e-f012535c3af4" (UID: "4a06ef38-4591-4855-837e-f012535c3af4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:24:17.366424 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.366408 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-registry-certificates\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:24:17.366506 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.366425 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4a06ef38-4591-4855-837e-f012535c3af4" (UID: "4a06ef38-4591-4855-837e-f012535c3af4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:24:17.366506 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.366431 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a06ef38-4591-4855-837e-f012535c3af4-ca-trust-extracted\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:24:17.368231 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.368207 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4a06ef38-4591-4855-837e-f012535c3af4" (UID: "4a06ef38-4591-4855-837e-f012535c3af4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:24:17.368335 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.368237 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4a06ef38-4591-4855-837e-f012535c3af4" (UID: "4a06ef38-4591-4855-837e-f012535c3af4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:24:17.368382 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.368372 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-kube-api-access-wv2f6" (OuterVolumeSpecName: "kube-api-access-wv2f6") pod "4a06ef38-4591-4855-837e-f012535c3af4" (UID: "4a06ef38-4591-4855-837e-f012535c3af4"). InnerVolumeSpecName "kube-api-access-wv2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:24:17.368417 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.368395 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4a06ef38-4591-4855-837e-f012535c3af4" (UID: "4a06ef38-4591-4855-837e-f012535c3af4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:24:17.467539 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.467513 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-bound-sa-token\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:24:17.467648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.467541 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-image-registry-private-configuration\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:24:17.467648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.467554 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2f6\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-kube-api-access-wv2f6\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:24:17.467648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.467565 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a06ef38-4591-4855-837e-f012535c3af4-trusted-ca\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:24:17.467648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.467575 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a06ef38-4591-4855-837e-f012535c3af4-installation-pull-secrets\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:24:17.597780 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:17.597716 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:18.176437 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:18.176351 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:18.285977 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:18.285942 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9c745db8b-dhqbq" Jul 06 12:24:18.286474 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:18.286436 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" event={"ID":"22ca1f39-285a-4d4b-953f-ae368fe5d4a8","Type":"ContainerStarted","Data":"a50f878a4ee7b83fcbcf821d62f6e49be64b4cd595070fb4bf21acd86ca83ea5"} Jul 06 12:24:18.373260 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:18.373222 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9c745db8b-dhqbq"] Jul 06 12:24:18.385812 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:18.385779 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9c745db8b-dhqbq"] Jul 06 12:24:18.475611 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:18.475585 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a06ef38-4591-4855-837e-f012535c3af4-registry-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:24:18.595211 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:18.595174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:18.954277 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:18.954202 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:19.138464 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:19.138423 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:19.514989 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:19.514956 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:19.830798 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:19.830719 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a06ef38-4591-4855-837e-f012535c3af4" path="/var/lib/kubelet/pods/4a06ef38-4591-4855-837e-f012535c3af4/volumes" Jul 06 12:24:19.901727 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:19.901696 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:20.090695 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:20.090600 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:20.190682 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:20.190631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:24:20.193292 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:20.193268 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0caf416-8c07-442a-8c58-f06069227b50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-sgkbc\" (UID: \"f0caf416-8c07-442a-8c58-f06069227b50\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:24:20.434555 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:20.434474 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jnrhh\"" Jul 06 12:24:20.441885 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:20.441858 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" Jul 06 12:24:20.726695 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:20.726653 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc"] Jul 06 12:24:20.734609 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:20.734579 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0caf416_8c07_442a_8c58_f06069227b50.slice/crio-31e2e32adf430bc7b889fc93d3f728f44123fb631d8aea40830bec9260f2e9e8 WatchSource:0}: Error finding container 31e2e32adf430bc7b889fc93d3f728f44123fb631d8aea40830bec9260f2e9e8: Status 404 returned error can't find the container with id 31e2e32adf430bc7b889fc93d3f728f44123fb631d8aea40830bec9260f2e9e8 Jul 06 12:24:21.298535 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:21.298501 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" event={"ID":"22ca1f39-285a-4d4b-953f-ae368fe5d4a8","Type":"ContainerStarted","Data":"ff2baf94ed176637ce52f8261bde7852c7622f869e3f07723882e92d093d772a"} Jul 06 12:24:21.298738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:21.298544 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" event={"ID":"22ca1f39-285a-4d4b-953f-ae368fe5d4a8","Type":"ContainerStarted","Data":"43c3bf5dace437eb9414f1f2fe4c931288c7641559c0b231e93c621d59ac739b"} Jul 06 12:24:21.298738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:21.298564 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" event={"ID":"22ca1f39-285a-4d4b-953f-ae368fe5d4a8","Type":"ContainerStarted","Data":"6748e47fc665d09c5af24278e2ee624a63dd2cbd1b3cda864298d1cbcbba5317"} Jul 06 12:24:21.299642 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:21.299618 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" event={"ID":"f0caf416-8c07-442a-8c58-f06069227b50","Type":"ContainerStarted","Data":"31e2e32adf430bc7b889fc93d3f728f44123fb631d8aea40830bec9260f2e9e8"} Jul 06 12:24:21.328622 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:21.328578 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c575594-h8jbp" podStartSLOduration=1.96725214 podStartE2EDuration="5.328563826s" totalImagesPullingTime="3.361311686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:21.327050763 +0000 UTC m=+77.405598790" watchObservedRunningTime="2026-07-06 12:24:21.328563826 +0000 UTC m=+77.407111849" Jul 06 12:24:22.668958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:22.668915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:22.786187 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:22.786140 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:22.877478 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:22.877441 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:22.980414 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:22.980383 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:23.308912 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:23.308878 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" event={"ID":"f0caf416-8c07-442a-8c58-f06069227b50","Type":"ContainerStarted","Data":"163abf4823224a572ab0426d45e5ec226e219b365ccb18f01be02f23b0126b4a"} Jul 06 12:24:23.367356 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:23.367306 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-sgkbc" podStartSLOduration=33.671155706 podStartE2EDuration="35.367290304s" totalImagesPullingTime="1.696134598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:23.366526951 +0000 UTC m=+79.445074978" watchObservedRunningTime="2026-07-06 12:24:23.367290304 +0000 UTC m=+79.445838331" Jul 06 12:24:23.466897 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:23.466864 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:24.323421 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:24.323393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:24:24.325532 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:24.325502 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c970c174-8210-443d-a38c-7542aeac0e1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrf5l\" (UID: \"c970c174-8210-443d-a38c-7542aeac0e1e\") " pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:24:24.617815 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:24.617738 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9zwn2\"" Jul 06 12:24:24.625196 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:24.625173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lrf5l" Jul 06 12:24:24.744996 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:24.744967 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lrf5l"] Jul 06 12:24:24.748224 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:24.748197 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc970c174_8210_443d_a38c_7542aeac0e1e.slice/crio-b253fcee1f97c3ca66c882497af7a8b725e4d278bbb316fc9ca9ac15fca7f92d WatchSource:0}: Error finding container b253fcee1f97c3ca66c882497af7a8b725e4d278bbb316fc9ca9ac15fca7f92d: Status 404 returned error can't find the container with id b253fcee1f97c3ca66c882497af7a8b725e4d278bbb316fc9ca9ac15fca7f92d Jul 06 12:24:25.318879 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:25.318796 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrf5l" event={"ID":"c970c174-8210-443d-a38c-7542aeac0e1e","Type":"ContainerStarted","Data":"b253fcee1f97c3ca66c882497af7a8b725e4d278bbb316fc9ca9ac15fca7f92d"} Jul 06 12:24:26.065914 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:26.065781 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:26.286914 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:26.286877 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:26.704977 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:26.704945 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:29.334233 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:29.334154 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrf5l" event={"ID":"c970c174-8210-443d-a38c-7542aeac0e1e","Type":"ContainerStarted","Data":"beb176572401e66d307a5e72ea479061bce0f1fcd1074704c89e91aa7582f40d"} Jul 06 12:24:29.334233 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:29.334207 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrf5l" event={"ID":"c970c174-8210-443d-a38c-7542aeac0e1e","Type":"ContainerStarted","Data":"0212b497e7d28f3e71eee66c09a6b9a66b8fd28b1795b66ea78282dc1a25730b"} Jul 06 12:24:30.067634 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:30.067602 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:32.240775 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.240745 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:32.345434 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.345401 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrf5l" event={"ID":"c970c174-8210-443d-a38c-7542aeac0e1e","Type":"ContainerStarted","Data":"44c346be938118f1f8de302cf4523973c429ad1f0fec34d9e9806209ed7ece30"} Jul 06 12:24:32.367111 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.367056 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lrf5l" podStartSLOduration=33.854028552 podStartE2EDuration="40.367041191s" totalImagesPullingTime="6.513012639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:23:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:32.366316343 +0000 UTC m=+88.444864382" watchObservedRunningTime="2026-07-06 12:24:32.367041191 +0000 UTC m=+88.445589196" Jul 06 12:24:32.504182 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.504109 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:32.576335 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.576302 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jj89d"] Jul 06 12:24:32.579600 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.579583 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.582466 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.582443 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 12:24:32.582768 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.582747 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9jcmc\"" Jul 06 12:24:32.582888 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.582764 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 12:24:32.582888 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.582747 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 12:24:32.582888 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.582785 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 12:24:32.613986 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.613961 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:32.692204 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.692173 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.692204 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.692205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-textfile\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.692384 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.692228 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-accelerators-collector-config\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.692384 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.692312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsxvl\" (UniqueName: \"kubernetes.io/projected/dade6ad6-8494-4c81-b45c-e14cc7b48e29-kube-api-access-gsxvl\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.692384 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.692348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-wtmp\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.692384 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.692377 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dade6ad6-8494-4c81-b45c-e14cc7b48e29-root\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.692524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.692429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dade6ad6-8494-4c81-b45c-e14cc7b48e29-sys\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.692524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.692448 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-tls\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.692524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.692470 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dade6ad6-8494-4c81-b45c-e14cc7b48e29-metrics-client-ca\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.697369 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.697345 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:32.793160 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793070 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsxvl\" (UniqueName: \"kubernetes.io/projected/dade6ad6-8494-4c81-b45c-e14cc7b48e29-kube-api-access-gsxvl\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793160 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793125 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-wtmp\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793161 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dade6ad6-8494-4c81-b45c-e14cc7b48e29-root\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793210 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dade6ad6-8494-4c81-b45c-e14cc7b48e29-sys\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793240 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-tls\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793261 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dade6ad6-8494-4c81-b45c-e14cc7b48e29-metrics-client-ca\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793278 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dade6ad6-8494-4c81-b45c-e14cc7b48e29-root\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793297 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dade6ad6-8494-4c81-b45c-e14cc7b48e29-sys\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793308 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793345 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-wtmp\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:32.793351 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793350 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-textfile\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:32.793433 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-tls podName:dade6ad6-8494-4c81-b45c-e14cc7b48e29 nodeName:}" failed. No retries permitted until 2026-07-06 12:24:33.293412671 +0000 UTC m=+89.371960682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-tls") pod "node-exporter-jj89d" (UID: "dade6ad6-8494-4c81-b45c-e14cc7b48e29") : secret "node-exporter-tls" not found Jul 06 12:24:32.793458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-accelerators-collector-config\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.793992 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793752 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-textfile\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.794034 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.793991 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dade6ad6-8494-4c81-b45c-e14cc7b48e29-metrics-client-ca\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.794104 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.794038 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-accelerators-collector-config\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.795636 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.795615 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.804494 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.804474 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsxvl\" (UniqueName: \"kubernetes.io/projected/dade6ad6-8494-4c81-b45c-e14cc7b48e29-kube-api-access-gsxvl\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:32.885931 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:32.885882 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:33.007407 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.007375 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:33.106876 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.106802 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:33.219057 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.219019 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:33.296639 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.296609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-tls\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:33.298891 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.298858 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dade6ad6-8494-4c81-b45c-e14cc7b48e29-node-exporter-tls\") pod \"node-exporter-jj89d\" (UID: \"dade6ad6-8494-4c81-b45c-e14cc7b48e29\") " pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:33.366214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.366151 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:33.447380 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.447353 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:33.488782 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.488758 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jj89d" Jul 06 12:24:33.501490 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:33.501468 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddade6ad6_8494_4c81_b45c_e14cc7b48e29.slice/crio-6e3aec0816fdb7000fa49116ed813d6cfd4f1e23e5e5ae3bb43863e653a2e2a7 WatchSource:0}: Error finding container 6e3aec0816fdb7000fa49116ed813d6cfd4f1e23e5e5ae3bb43863e653a2e2a7: Status 404 returned error can't find the container with id 6e3aec0816fdb7000fa49116ed813d6cfd4f1e23e5e5ae3bb43863e653a2e2a7 Jul 06 12:24:33.552129 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.552106 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:33.652802 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.652734 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 12:24:33.657288 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.657272 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.658931 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.658910 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:33.663213 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.663194 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 12:24:33.664721 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.664706 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 12:24:33.665264 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.665206 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rg7qn\"" Jul 06 12:24:33.665264 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.665210 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 12:24:33.665264 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.665241 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 12:24:33.665264 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.665252 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 12:24:33.665599 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.665232 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 12:24:33.665599 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.665210 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 12:24:33.665599 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.665558 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 12:24:33.665785 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.665695 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 12:24:33.696110 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.696088 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 12:24:33.801283 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-out\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801467 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801313 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801467 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801467 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801402 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801467 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801670 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801470 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rls2t\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-kube-api-access-rls2t\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801670 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801498 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801670 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801526 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801670 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-web-config\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801864 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801673 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801864 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801711 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801864 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801738 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.801864 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.801797 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.902597 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.902559 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rls2t\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-kube-api-access-rls2t\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.902831 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.902611 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.902831 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.902637 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.902831 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.902737 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-web-config\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.902831 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.902769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.903086 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.902890 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.903086 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.902928 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.903086 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.902994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.903086 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.903054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-out\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.903086 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.903077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.903335 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:33.903100 2572 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 12:24:33.903335 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.903129 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.903335 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.903129 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.903335 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:24:33.903168 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-main-tls podName:1b32201d-892d-40dd-8f5e-fe8962e25a73 nodeName:}" failed. No retries permitted until 2026-07-06 12:24:34.403147562 +0000 UTC m=+90.481695588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73") : secret "alertmanager-main-tls" not found Jul 06 12:24:33.903335 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.903218 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.903335 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.903255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.905286 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.904218 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.905286 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.905063 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.906800 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.906756 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.906908 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.906893 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.907022 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.906981 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.907586 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.907532 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-out\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.907586 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.907572 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.907756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.907702 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-web-config\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.908199 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.908177 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.909392 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.909369 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.913101 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.913073 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rls2t\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-kube-api-access-rls2t\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:33.929399 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:33.929371 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:34.024518 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:34.024476 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:34.140139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:34.140105 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:34.355247 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:34.355199 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jj89d" event={"ID":"dade6ad6-8494-4c81-b45c-e14cc7b48e29","Type":"ContainerStarted","Data":"6e3aec0816fdb7000fa49116ed813d6cfd4f1e23e5e5ae3bb43863e653a2e2a7"} Jul 06 12:24:34.407370 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:34.407343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:34.409736 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:34.409686 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:34.566420 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:34.566388 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:24:34.711506 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:34.711476 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 12:24:34.713514 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:34.713495 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b32201d_892d_40dd_8f5e_fe8962e25a73.slice/crio-bc357f159c83c27c3c78c8c64c5a0e35672c34a39b075f83b22036b8e61592c4 WatchSource:0}: Error finding container bc357f159c83c27c3c78c8c64c5a0e35672c34a39b075f83b22036b8e61592c4: Status 404 returned error can't find the container with id bc357f159c83c27c3c78c8c64c5a0e35672c34a39b075f83b22036b8e61592c4 Jul 06 12:24:35.108124 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:35.108088 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:35.322326 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:35.322294 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:35.360330 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:35.360251 2572 generic.go:356] "Generic (PLEG): container finished" podID="dade6ad6-8494-4c81-b45c-e14cc7b48e29" containerID="5c5ce2bd8425e08268d27702dce5bae9112481c68bd71f56601fa242ce193421" exitCode=0 Jul 06 12:24:35.360755 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:35.360349 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jj89d" event={"ID":"dade6ad6-8494-4c81-b45c-e14cc7b48e29","Type":"ContainerDied","Data":"5c5ce2bd8425e08268d27702dce5bae9112481c68bd71f56601fa242ce193421"} Jul 06 12:24:35.361847 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:35.361819 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerStarted","Data":"bc357f159c83c27c3c78c8c64c5a0e35672c34a39b075f83b22036b8e61592c4"} Jul 06 12:24:35.547472 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:35.547438 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:35.694034 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:35.693951 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:35.906696 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:35.906669 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:36.018817 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:36.018786 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:36.125252 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:36.125226 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:36.367247 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:36.367160 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jj89d" event={"ID":"dade6ad6-8494-4c81-b45c-e14cc7b48e29","Type":"ContainerStarted","Data":"0f6828c8289552ad7d5e12fde8cbecba16868e6fac52585060e948630ab8c01b"} Jul 06 12:24:36.367247 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:36.367224 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jj89d" event={"ID":"dade6ad6-8494-4c81-b45c-e14cc7b48e29","Type":"ContainerStarted","Data":"63da63e4aa5145e6120d43e792f24c052cef23553a615012a20241bdae43c5df"} Jul 06 12:24:36.368709 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:36.368683 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerID="a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069" exitCode=0 Jul 06 12:24:36.368809 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:36.368719 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerDied","Data":"a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069"} Jul 06 12:24:36.392932 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:36.392893 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jj89d" podStartSLOduration=3.423056217 podStartE2EDuration="4.392880458s" totalImagesPullingTime="905.563619ms" totalInitContainerRuntime="64.260622ms" isStatefulPod=true podCreationTimestamp="2026-07-06 12:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:36.391321348 +0000 UTC m=+92.469869374" watchObservedRunningTime="2026-07-06 12:24:36.392880458 +0000 UTC m=+92.471428485" Jul 06 12:24:36.414639 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:36.414614 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:37.287958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.287911 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:37.363955 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.363925 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:37.837075 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.837048 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8"] Jul 06 12:24:37.840049 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.840027 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:37.845796 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.845747 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 12:24:37.845919 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.845844 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 12:24:37.845919 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.845854 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 12:24:37.846967 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.846950 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7w6rt\"" Jul 06 12:24:37.863498 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.863481 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 12:24:37.863593 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.863486 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 12:24:37.863633 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.863586 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 12:24:37.874418 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.874396 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8"] Jul 06 12:24:37.934955 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.934934 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-telemeter-client-tls\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:37.935039 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.934963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:37.935039 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.934987 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/523e67d1-8f4e-4fa2-8421-c083dc84dd64-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:37.935128 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.935111 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/523e67d1-8f4e-4fa2-8421-c083dc84dd64-metrics-client-ca\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:37.935175 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.935162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-secret-telemeter-client\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:37.935283 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.935259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdk2\" (UniqueName: \"kubernetes.io/projected/523e67d1-8f4e-4fa2-8421-c083dc84dd64-kube-api-access-lpdk2\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:37.935326 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.935304 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/523e67d1-8f4e-4fa2-8421-c083dc84dd64-serving-certs-ca-bundle\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:37.935363 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:37.935353 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-federate-client-tls\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.000610 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.000570 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:38.036182 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.036156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/523e67d1-8f4e-4fa2-8421-c083dc84dd64-metrics-client-ca\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.036297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.036220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-secret-telemeter-client\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.036297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.036274 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdk2\" (UniqueName: \"kubernetes.io/projected/523e67d1-8f4e-4fa2-8421-c083dc84dd64-kube-api-access-lpdk2\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.036405 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.036304 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/523e67d1-8f4e-4fa2-8421-c083dc84dd64-serving-certs-ca-bundle\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.036405 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.036339 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-federate-client-tls\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.036405 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.036370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-telemeter-client-tls\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.036567 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.036398 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.036567 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.036432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/523e67d1-8f4e-4fa2-8421-c083dc84dd64-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.037111 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.036943 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/523e67d1-8f4e-4fa2-8421-c083dc84dd64-metrics-client-ca\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.037351 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.037284 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/523e67d1-8f4e-4fa2-8421-c083dc84dd64-serving-certs-ca-bundle\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.037351 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.037284 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/523e67d1-8f4e-4fa2-8421-c083dc84dd64-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.039301 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.039281 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-telemeter-client-tls\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.039442 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.039422 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-federate-client-tls\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.039910 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.039893 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-secret-telemeter-client\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.040013 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.039992 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/523e67d1-8f4e-4fa2-8421-c083dc84dd64-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.052992 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.052967 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdk2\" (UniqueName: \"kubernetes.io/projected/523e67d1-8f4e-4fa2-8421-c083dc84dd64-kube-api-access-lpdk2\") pod \"telemeter-client-9c9fcffb4-rfpg8\" (UID: \"523e67d1-8f4e-4fa2-8421-c083dc84dd64\") " pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.149291 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.149254 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" Jul 06 12:24:38.285756 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.285728 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8"] Jul 06 12:24:38.286284 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:38.286263 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523e67d1_8f4e_4fa2_8421_c083dc84dd64.slice/crio-e17a82c6906b8963b4f2b87146891522bfe25795e9d72c199d6302e59762e2df WatchSource:0}: Error finding container e17a82c6906b8963b4f2b87146891522bfe25795e9d72c199d6302e59762e2df: Status 404 returned error can't find the container with id e17a82c6906b8963b4f2b87146891522bfe25795e9d72c199d6302e59762e2df Jul 06 12:24:38.377920 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.377859 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerStarted","Data":"7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf"} Jul 06 12:24:38.378016 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.377932 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerStarted","Data":"8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3"} Jul 06 12:24:38.378016 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.377971 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerStarted","Data":"94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38"} Jul 06 12:24:38.378016 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.377998 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerStarted","Data":"d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7"} Jul 06 12:24:38.378139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.378021 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerStarted","Data":"75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1"} Jul 06 12:24:38.378980 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.378962 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" event={"ID":"523e67d1-8f4e-4fa2-8421-c083dc84dd64","Type":"ContainerStarted","Data":"e17a82c6906b8963b4f2b87146891522bfe25795e9d72c199d6302e59762e2df"} Jul 06 12:24:38.845792 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.845758 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:38.850472 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.850449 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 12:24:38.853875 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.853862 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.857902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.857850 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 12:24:38.858019 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.857923 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l76dc\"" Jul 06 12:24:38.858019 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.857850 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 12:24:38.858509 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.858364 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1gt5i394uragt\"" Jul 06 12:24:38.858509 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.858375 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 12:24:38.858509 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.858443 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 12:24:38.858722 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.858587 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 12:24:38.858774 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.858730 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 12:24:38.861884 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.861863 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 12:24:38.861884 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.861873 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 12:24:38.862014 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.861884 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 12:24:38.862691 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.862641 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 12:24:38.862774 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.862759 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 12:24:38.862893 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.862874 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 12:24:38.863214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.863199 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 12:24:38.866520 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.866498 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 12:24:38.869419 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.869397 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 12:24:38.875799 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.875782 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 12:24:38.944559 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944531 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.944653 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944600 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.944742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944652 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.944742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944698 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.944742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.944903 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944765 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.944903 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.944903 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944839 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.944903 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945102 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.944966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945102 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945006 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945102 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945102 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945063 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945102 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945093 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945348 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945195 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945348 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945224 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbs9\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-kube-api-access-4tbs9\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945348 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945348 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945310 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945348 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:38.945589 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:38.945397 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.045949 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.045915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.045949 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.045950 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046130 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.045981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046130 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046111 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046213 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046151 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046213 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046213 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046201 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046350 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046224 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046350 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046350 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbs9\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-kube-api-access-4tbs9\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046350 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046533 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046374 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046533 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046533 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046533 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046475 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046533 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046520 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046575 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.046886 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046858 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.047135 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.046933 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.047597 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.047230 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.048007 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.047893 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.048007 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.047930 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.049237 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.049210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.050108 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.049632 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.050108 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.049763 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.050108 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.049777 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.050108 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.049985 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.050108 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.050069 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.050510 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.050488 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.051031 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.051003 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.051120 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.051067 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.051334 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.051313 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.051584 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.051568 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.052247 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.052226 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.052358 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.052335 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.052652 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.052635 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.056854 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.056830 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbs9\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-kube-api-access-4tbs9\") pod \"prometheus-k8s-0\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.168967 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.168883 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:39.324091 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:39.324059 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0370a1_7056_41a3_a854_a90c5fea7c2b.slice/crio-f1ba08a0e7c68ba284c6986076a8097c02190c2d4b3e0ad198657c0b8f5159c8 WatchSource:0}: Error finding container f1ba08a0e7c68ba284c6986076a8097c02190c2d4b3e0ad198657c0b8f5159c8: Status 404 returned error can't find the container with id f1ba08a0e7c68ba284c6986076a8097c02190c2d4b3e0ad198657c0b8f5159c8 Jul 06 12:24:39.324636 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.324611 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 12:24:39.388373 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.388338 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerStarted","Data":"e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac"} Jul 06 12:24:39.389945 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.389918 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerStarted","Data":"f1ba08a0e7c68ba284c6986076a8097c02190c2d4b3e0ad198657c0b8f5159c8"} Jul 06 12:24:39.418804 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.418758 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.107187862 podStartE2EDuration="6.418740646s" totalImagesPullingTime="3.296422716s" totalInitContainerRuntime="15.130068ms" isStatefulPod=false podCreationTimestamp="2026-07-06 12:24:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:39.416759456 +0000 UTC m=+95.495307483" watchObservedRunningTime="2026-07-06 12:24:39.418740646 +0000 UTC m=+95.497288695" Jul 06 12:24:39.430867 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.430800 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:39.527435 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:39.527409 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:40.395823 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:40.395707 2572 generic.go:356] "Generic (PLEG): container finished" podID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerID="cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229" exitCode=0 Jul 06 12:24:40.395823 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:40.395788 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerDied","Data":"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229"} Jul 06 12:24:40.398160 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:40.398138 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" event={"ID":"523e67d1-8f4e-4fa2-8421-c083dc84dd64","Type":"ContainerStarted","Data":"32c1495abc916ed0183f863f52c372cf7e98e9f919701fdf8b0d04ec160d9817"} Jul 06 12:24:41.405405 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:41.405370 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" event={"ID":"523e67d1-8f4e-4fa2-8421-c083dc84dd64","Type":"ContainerStarted","Data":"c1f04e0c367428b9059dbc10f753e4d3f7f4abefa92598e5ff73ea7c248a03d0"} Jul 06 12:24:41.405892 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:41.405445 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" event={"ID":"523e67d1-8f4e-4fa2-8421-c083dc84dd64","Type":"ContainerStarted","Data":"c6b9d0fe13412b5db9e6cfa945ba84c73f686d6323bbaa2231262fec49dbd450"} Jul 06 12:24:41.441952 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:41.441902 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-9c9fcffb4-rfpg8" podStartSLOduration=2.4608191440000002 podStartE2EDuration="4.441888803s" totalImagesPullingTime="1.981069659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:41.440882255 +0000 UTC m=+97.519430281" watchObservedRunningTime="2026-07-06 12:24:41.441888803 +0000 UTC m=+97.520436830" Jul 06 12:24:42.742779 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:42.742694 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:42.843274 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:42.843238 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:43.794786 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:43.794753 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:24:43.798233 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:43.798017 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8b90c0d-5b63-442c-add0-54a38a3c435e-metrics-tls\") pod \"dns-default-hmvrc\" (UID: \"a8b90c0d-5b63-442c-add0-54a38a3c435e\") " pod="openshift-dns/dns-default-hmvrc" Jul 06 12:24:44.031384 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.031338 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:44.063809 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.063740 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gt55k\"" Jul 06 12:24:44.071035 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.071012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmvrc" Jul 06 12:24:44.147457 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.147362 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:44.224718 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.224690 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmvrc"] Jul 06 12:24:44.227615 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:24:44.227592 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b90c0d_5b63_442c_add0_54a38a3c435e.slice/crio-a1f185697ef4d11862dd9d2741fae76254ea432fb206b0a34085019ce08c98c4 WatchSource:0}: Error finding container a1f185697ef4d11862dd9d2741fae76254ea432fb206b0a34085019ce08c98c4: Status 404 returned error can't find the container with id a1f185697ef4d11862dd9d2741fae76254ea432fb206b0a34085019ce08c98c4 Jul 06 12:24:44.418221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.418122 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerStarted","Data":"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484"} Jul 06 12:24:44.418221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.418210 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerStarted","Data":"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d"} Jul 06 12:24:44.419436 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.419399 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmvrc" event={"ID":"a8b90c0d-5b63-442c-add0-54a38a3c435e","Type":"ContainerStarted","Data":"a1f185697ef4d11862dd9d2741fae76254ea432fb206b0a34085019ce08c98c4"} Jul 06 12:24:44.846520 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.846481 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:44.949189 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:44.949139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:45.039918 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:45.039889 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:46.310291 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:46.310261 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:46.410691 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:46.410647 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:47.283454 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.283429 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-scpmd" Jul 06 12:24:47.433765 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.433729 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerStarted","Data":"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674"} Jul 06 12:24:47.434138 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.433809 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerStarted","Data":"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308"} Jul 06 12:24:47.434138 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.433854 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerStarted","Data":"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d"} Jul 06 12:24:47.434138 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.433895 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerStarted","Data":"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408"} Jul 06 12:24:47.435487 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.435465 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmvrc" event={"ID":"a8b90c0d-5b63-442c-add0-54a38a3c435e","Type":"ContainerStarted","Data":"b1ace4db7682777ee44f03eac1bc16379f74acf02cf20580104bffbd459d6076"} Jul 06 12:24:47.435598 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.435502 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmvrc" event={"ID":"a8b90c0d-5b63-442c-add0-54a38a3c435e","Type":"ContainerStarted","Data":"ca1fa396857340fa0b455ac96d0a34c388c705168d6f5a4ec03162cd5591525d"} Jul 06 12:24:47.435639 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.435606 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hmvrc" Jul 06 12:24:47.486153 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.486101 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.383507282 podStartE2EDuration="9.486084269s" totalImagesPullingTime="6.085207136s" totalInitContainerRuntime="17.369851ms" isStatefulPod=false podCreationTimestamp="2026-07-06 12:24:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:47.484792324 +0000 UTC m=+103.563340351" watchObservedRunningTime="2026-07-06 12:24:47.486084269 +0000 UTC m=+103.564632295" Jul 06 12:24:47.525703 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:47.525620 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hmvrc" podStartSLOduration=66.136693439 podStartE2EDuration="1m8.525607455s" totalImagesPullingTime="2.388914016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:24:47.523021927 +0000 UTC m=+103.601569954" watchObservedRunningTime="2026-07-06 12:24:47.525607455 +0000 UTC m=+103.604155482" Jul 06 12:24:48.573530 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:48.573503 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:48.721953 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:48.721923 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:49.169615 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:49.169585 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:24:52.333380 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:52.333347 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:52.558118 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:52.558077 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:52.844863 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:52.844830 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:53.030815 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:53.030783 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:54.667320 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:54.667291 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:54.774863 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:54.774836 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:55.497248 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:55.497215 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:55.606701 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:55.606646 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:56.566738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:56.566707 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:56.660684 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:56.660639 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:57.165132 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:57.165099 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:57.273717 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:57.273682 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:57.380017 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:57.379986 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:24:57.442647 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:24:57.442582 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hmvrc" Jul 06 12:25:01.633897 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:01.633860 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:01.799378 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:01.799347 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:02.489074 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:02.489042 2572 generic.go:356] "Generic (PLEG): container finished" podID="9b696dd3-c3e9-47c5-b1fe-366497f5b1e3" containerID="3ca3487f3916989730eca89b87feef44f609a9b2f071027d762d50b878be1285" exitCode=0 Jul 06 12:25:02.489235 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:02.489110 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" event={"ID":"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3","Type":"ContainerDied","Data":"3ca3487f3916989730eca89b87feef44f609a9b2f071027d762d50b878be1285"} Jul 06 12:25:02.489447 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:02.489434 2572 scope.go:117] "RemoveContainer" containerID="3ca3487f3916989730eca89b87feef44f609a9b2f071027d762d50b878be1285" Jul 06 12:25:02.918441 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:02.918360 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:02.999789 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:02.999759 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:03.494688 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:03.494636 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qdm6x" event={"ID":"9b696dd3-c3e9-47c5-b1fe-366497f5b1e3","Type":"ContainerStarted","Data":"e86062979480778cc3e32ffe5fb1d2c43e5629bfe4f508fc3f39ffb3b48a849c"} Jul 06 12:25:03.733682 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:03.733629 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:03.865196 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:03.865124 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:04.274250 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:04.274220 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:04.441344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:04.441314 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:06.729681 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:06.729636 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:06.837846 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:06.837816 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:08.176001 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:08.175970 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:09.700893 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:09.700854 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:09.830442 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:09.830411 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:14.649340 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:14.649307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:25:14.651585 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:14.651548 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2598c5d0-e3bb-4604-bd89-064fcf5e0319-metrics-certs\") pod \"network-metrics-daemon-f5mqz\" (UID: \"2598c5d0-e3bb-4604-bd89-064fcf5e0319\") " pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:25:14.658190 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:14.658170 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l5tqd\"" Jul 06 12:25:14.664924 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:14.664896 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5mqz" Jul 06 12:25:14.797955 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:14.797924 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f5mqz"] Jul 06 12:25:14.801277 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:25:14.801250 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2598c5d0_e3bb_4604_bd89_064fcf5e0319.slice/crio-b4febad19ff63c9c62b48c39b60cc60954cbeb19d8fbf3624195931e49090640 WatchSource:0}: Error finding container b4febad19ff63c9c62b48c39b60cc60954cbeb19d8fbf3624195931e49090640: Status 404 returned error can't find the container with id b4febad19ff63c9c62b48c39b60cc60954cbeb19d8fbf3624195931e49090640 Jul 06 12:25:15.542840 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:15.542804 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f5mqz" event={"ID":"2598c5d0-e3bb-4604-bd89-064fcf5e0319","Type":"ContainerStarted","Data":"b4febad19ff63c9c62b48c39b60cc60954cbeb19d8fbf3624195931e49090640"} Jul 06 12:25:16.548474 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:16.548436 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f5mqz" event={"ID":"2598c5d0-e3bb-4604-bd89-064fcf5e0319","Type":"ContainerStarted","Data":"251bd4ee208afb559450d36c1f4765fc2bbff04de714abaf6de6dde5ff1d2963"} Jul 06 12:25:16.548881 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:16.548505 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f5mqz" event={"ID":"2598c5d0-e3bb-4604-bd89-064fcf5e0319","Type":"ContainerStarted","Data":"3acc9c5f7971934b22f77b8ac39470544009d5e9a116011fce26de7e8bd915b0"} Jul 06 12:25:16.568026 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:16.567982 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f5mqz" podStartSLOduration=129.513988187 podStartE2EDuration="2m10.567966505s" totalImagesPullingTime="1.053978318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:25:16.566685866 +0000 UTC m=+132.645233894" watchObservedRunningTime="2026-07-06 12:25:16.567966505 +0000 UTC m=+132.646514532" Jul 06 12:25:16.743607 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:16.743564 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:16.814368 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:16.814288 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:20.192134 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:20.192103 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:28.714552 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:28.714510 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:39.170013 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:39.169971 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:39.189789 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:39.189760 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:39.649167 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:39.649141 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:48.725108 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:48.725078 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:48.822927 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:48.822893 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:50.065042 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:50.065002 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:50.162385 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:50.162358 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:50.718278 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:50.718242 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:25:53.023203 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.023168 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 12:25:53.023708 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.023566 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="alertmanager" containerID="cri-o://75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1" gracePeriod=120 Jul 06 12:25:53.023708 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.023637 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="prom-label-proxy" containerID="cri-o://e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac" gracePeriod=120 Jul 06 12:25:53.023708 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.023675 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy-web" containerID="cri-o://94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38" gracePeriod=120 Jul 06 12:25:53.023888 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.023710 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy" containerID="cri-o://8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3" gracePeriod=120 Jul 06 12:25:53.023888 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.023681 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="config-reloader" containerID="cri-o://d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7" gracePeriod=120 Jul 06 12:25:53.023888 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.023613 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy-metric" containerID="cri-o://7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf" gracePeriod=120 Jul 06 12:25:53.688507 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.688478 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerID="e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac" exitCode=0 Jul 06 12:25:53.688507 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.688501 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerID="8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3" exitCode=0 Jul 06 12:25:53.688507 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.688507 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerID="d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7" exitCode=0 Jul 06 12:25:53.688507 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.688513 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerID="75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1" exitCode=0 Jul 06 12:25:53.688792 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.688553 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerDied","Data":"e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac"} Jul 06 12:25:53.688792 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.688604 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerDied","Data":"8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3"} Jul 06 12:25:53.688792 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.688632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerDied","Data":"d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7"} Jul 06 12:25:53.688792 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:53.688669 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerDied","Data":"75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1"} Jul 06 12:25:54.262979 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.262958 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:25:54.366375 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366302 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366375 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366340 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rls2t\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-kube-api-access-rls2t\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366385 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366415 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-cluster-tls-config\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366448 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-out\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366477 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-web\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366504 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-tls-assets\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366535 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-web-config\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366576 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-volume\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366964 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366606 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-main-db\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366964 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366652 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-metrics-client-ca\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366964 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366712 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-trusted-ca-bundle\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.366964 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.366740 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-main-tls\") pod \"1b32201d-892d-40dd-8f5e-fe8962e25a73\" (UID: \"1b32201d-892d-40dd-8f5e-fe8962e25a73\") " Jul 06 12:25:54.369351 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.369324 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-kube-api-access-rls2t" (OuterVolumeSpecName: "kube-api-access-rls2t") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "kube-api-access-rls2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:25:54.369606 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.369578 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:54.369736 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.369694 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:25:54.369891 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.369867 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:54.369989 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.369941 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:54.370078 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.369976 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:25:54.370158 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.370012 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:25:54.370312 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.370284 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:25:54.370395 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.370304 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:54.371351 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.371320 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-out" (OuterVolumeSpecName: "config-out") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:25:54.371796 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.371774 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-volume" (OuterVolumeSpecName: "config-volume") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:54.373123 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.373100 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:54.379674 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.379640 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-web-config" (OuterVolumeSpecName: "web-config") pod "1b32201d-892d-40dd-8f5e-fe8962e25a73" (UID: "1b32201d-892d-40dd-8f5e-fe8962e25a73"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:54.467377 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467351 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467377 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467376 2572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-cluster-tls-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467386 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-out\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467395 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467403 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-tls-assets\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467411 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-web-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467418 2572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-config-volume\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467425 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-main-db\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467434 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-metrics-client-ca\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467442 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b32201d-892d-40dd-8f5e-fe8962e25a73-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467450 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-main-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467458 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b32201d-892d-40dd-8f5e-fe8962e25a73-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.467505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.467468 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rls2t\" (UniqueName: \"kubernetes.io/projected/1b32201d-892d-40dd-8f5e-fe8962e25a73-kube-api-access-rls2t\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:54.695404 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.695321 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerID="7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf" exitCode=0 Jul 06 12:25:54.695404 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.695346 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerID="94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38" exitCode=0 Jul 06 12:25:54.695560 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.695409 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 12:25:54.695560 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.695413 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerDied","Data":"7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf"} Jul 06 12:25:54.695560 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.695522 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerDied","Data":"94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38"} Jul 06 12:25:54.695560 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.695554 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b32201d-892d-40dd-8f5e-fe8962e25a73","Type":"ContainerDied","Data":"bc357f159c83c27c3c78c8c64c5a0e35672c34a39b075f83b22036b8e61592c4"} Jul 06 12:25:54.695729 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.695593 2572 scope.go:117] "RemoveContainer" containerID="e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac" Jul 06 12:25:54.703551 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.703535 2572 scope.go:117] "RemoveContainer" containerID="7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf" Jul 06 12:25:54.710370 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.710352 2572 scope.go:117] "RemoveContainer" containerID="8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3" Jul 06 12:25:54.718475 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.718442 2572 scope.go:117] "RemoveContainer" containerID="94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38" Jul 06 12:25:54.720570 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.720552 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 12:25:54.726185 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.726164 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 12:25:54.728301 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.728288 2572 scope.go:117] "RemoveContainer" containerID="d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7" Jul 06 12:25:54.734906 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.734889 2572 scope.go:117] "RemoveContainer" containerID="75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1" Jul 06 12:25:54.741625 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.741607 2572 scope.go:117] "RemoveContainer" containerID="a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069" Jul 06 12:25:54.748462 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.748447 2572 scope.go:117] "RemoveContainer" containerID="e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac" Jul 06 12:25:54.748724 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:54.748704 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac\": container with ID starting with e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac not found: ID does not exist" containerID="e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac" Jul 06 12:25:54.748773 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.748733 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac"} err="failed to get container status \"e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac\": rpc error: code = NotFound desc = could not find container \"e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac\": container with ID starting with e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac not found: ID does not exist" Jul 06 12:25:54.748773 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.748764 2572 scope.go:117] "RemoveContainer" containerID="7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf" Jul 06 12:25:54.749002 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:54.748986 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf\": container with ID starting with 7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf not found: ID does not exist" containerID="7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf" Jul 06 12:25:54.749055 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749004 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf"} err="failed to get container status \"7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf\": rpc error: code = NotFound desc = could not find container \"7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf\": container with ID starting with 7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf not found: ID does not exist" Jul 06 12:25:54.749055 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749016 2572 scope.go:117] "RemoveContainer" containerID="8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3" Jul 06 12:25:54.749233 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:54.749217 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3\": container with ID starting with 8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3 not found: ID does not exist" containerID="8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3" Jul 06 12:25:54.749273 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749236 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3"} err="failed to get container status \"8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3\": rpc error: code = NotFound desc = could not find container \"8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3\": container with ID starting with 8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3 not found: ID does not exist" Jul 06 12:25:54.749273 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749249 2572 scope.go:117] "RemoveContainer" containerID="94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38" Jul 06 12:25:54.749455 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:54.749439 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38\": container with ID starting with 94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38 not found: ID does not exist" containerID="94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38" Jul 06 12:25:54.749501 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749458 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38"} err="failed to get container status \"94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38\": rpc error: code = NotFound desc = could not find container \"94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38\": container with ID starting with 94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38 not found: ID does not exist" Jul 06 12:25:54.749501 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749472 2572 scope.go:117] "RemoveContainer" containerID="d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7" Jul 06 12:25:54.749650 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:54.749634 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7\": container with ID starting with d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7 not found: ID does not exist" containerID="d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7" Jul 06 12:25:54.749704 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749653 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7"} err="failed to get container status \"d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7\": rpc error: code = NotFound desc = could not find container \"d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7\": container with ID starting with d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7 not found: ID does not exist" Jul 06 12:25:54.749704 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749690 2572 scope.go:117] "RemoveContainer" containerID="75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1" Jul 06 12:25:54.749902 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:54.749884 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1\": container with ID starting with 75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1 not found: ID does not exist" containerID="75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1" Jul 06 12:25:54.749952 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749904 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1"} err="failed to get container status \"75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1\": rpc error: code = NotFound desc = could not find container \"75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1\": container with ID starting with 75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1 not found: ID does not exist" Jul 06 12:25:54.749952 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.749915 2572 scope.go:117] "RemoveContainer" containerID="a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069" Jul 06 12:25:54.750108 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:54.750092 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069\": container with ID starting with a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069 not found: ID does not exist" containerID="a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069" Jul 06 12:25:54.750147 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750111 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069"} err="failed to get container status \"a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069\": rpc error: code = NotFound desc = could not find container \"a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069\": container with ID starting with a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069 not found: ID does not exist" Jul 06 12:25:54.750147 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750122 2572 scope.go:117] "RemoveContainer" containerID="e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac" Jul 06 12:25:54.750286 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750271 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac"} err="failed to get container status \"e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac\": rpc error: code = NotFound desc = could not find container \"e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac\": container with ID starting with e46727be9bbffb2f936c1cfe725b3a196d6f155aecc5e4adff983c952f0d34ac not found: ID does not exist" Jul 06 12:25:54.750286 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750285 2572 scope.go:117] "RemoveContainer" containerID="7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf" Jul 06 12:25:54.750444 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750421 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf"} err="failed to get container status \"7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf\": rpc error: code = NotFound desc = could not find container \"7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf\": container with ID starting with 7c147abb3e87be4dfedefbc85e28e50e04ff5082198e1b8bb51d01e2bd8f6ccf not found: ID does not exist" Jul 06 12:25:54.750482 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750446 2572 scope.go:117] "RemoveContainer" containerID="8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3" Jul 06 12:25:54.750694 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750674 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3"} err="failed to get container status \"8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3\": rpc error: code = NotFound desc = could not find container \"8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3\": container with ID starting with 8570d04f5a8974bc533440f6bd517aeccfa0767f1fad62e42b2f9016e76bf7c3 not found: ID does not exist" Jul 06 12:25:54.750753 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750696 2572 scope.go:117] "RemoveContainer" containerID="94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38" Jul 06 12:25:54.750958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750940 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38"} err="failed to get container status \"94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38\": rpc error: code = NotFound desc = could not find container \"94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38\": container with ID starting with 94758ad1380dc1e4f6b84802b51ff2ece221622cc5601c5a7b7d3dddc3f15d38 not found: ID does not exist" Jul 06 12:25:54.751006 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.750961 2572 scope.go:117] "RemoveContainer" containerID="d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7" Jul 06 12:25:54.751193 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.751174 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7"} err="failed to get container status \"d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7\": rpc error: code = NotFound desc = could not find container \"d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7\": container with ID starting with d9a234a800a81aaf2e598e8fe2b7a8073a8fca4f23a9789dbf9d6015a19985b7 not found: ID does not exist" Jul 06 12:25:54.751193 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.751189 2572 scope.go:117] "RemoveContainer" containerID="75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1" Jul 06 12:25:54.751500 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.751461 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1"} err="failed to get container status \"75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1\": rpc error: code = NotFound desc = could not find container \"75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1\": container with ID starting with 75559c137cb59d564964b5990a0587a79e2c83296397c2ef91faea968ad6e1a1 not found: ID does not exist" Jul 06 12:25:54.751500 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.751501 2572 scope.go:117] "RemoveContainer" containerID="a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069" Jul 06 12:25:54.751871 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:54.751842 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069"} err="failed to get container status \"a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069\": rpc error: code = NotFound desc = could not find container \"a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069\": container with ID starting with a8afd435d63c1b5f07224daa29757b637b430ae34387445b9dd1fb3b27fda069 not found: ID does not exist" Jul 06 12:25:55.830676 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:55.830627 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" path="/var/lib/kubelet/pods/1b32201d-892d-40dd-8f5e-fe8962e25a73/volumes" Jul 06 12:25:57.331776 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.331742 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 12:25:57.332198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.332018 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="prometheus" containerID="cri-o://e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" gracePeriod=600 Jul 06 12:25:57.332198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.332064 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="thanos-sidecar" containerID="cri-o://e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" gracePeriod=600 Jul 06 12:25:57.332198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.332084 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="config-reloader" containerID="cri-o://e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" gracePeriod=600 Jul 06 12:25:57.332198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.332118 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy-web" containerID="cri-o://1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" gracePeriod=600 Jul 06 12:25:57.332198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.332064 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy" containerID="cri-o://a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" gracePeriod=600 Jul 06 12:25:57.332198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.332090 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" gracePeriod=600 Jul 06 12:25:57.586648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.586593 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.690149 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690121 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-kube-rbac-proxy\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690318 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690157 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-db\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690318 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690179 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-metrics-client-certs\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690318 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690200 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-kubelet-serving-ca-bundle\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690318 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690218 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-2\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690318 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690235 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-web-config\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690318 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690259 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tbs9\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-kube-api-access-4tbs9\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690318 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690286 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690318 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690315 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-tls-assets\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690339 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-0\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690375 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-grpc-tls\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690406 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-serving-certs-ca-bundle\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690431 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-tls\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690467 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-trusted-ca-bundle\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690500 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690538 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-thanos-prometheus-http-client-file\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690551 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690566 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-1\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690597 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690649 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config-out\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.690749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690699 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.691344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690729 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-metrics-client-ca\") pod \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\" (UID: \"0b0370a1-7056-41a3-a854-a90c5fea7c2b\") " Jul 06 12:25:57.691344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.690850 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:25:57.691344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.691069 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.691344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.691087 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.691344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.691105 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.691344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.691130 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:25:57.691655 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.691390 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:25:57.692131 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.691913 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:25:57.692989 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.692940 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:25:57.693768 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.693738 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:57.694113 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.694071 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:25:57.694220 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.694189 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config" (OuterVolumeSpecName: "config") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:57.694448 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.694425 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config-out" (OuterVolumeSpecName: "config-out") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:25:57.694627 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.694585 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:25:57.694747 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.694713 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:57.694811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.694756 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:57.694925 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.694890 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:57.695091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.695064 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:57.695186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.695096 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:57.695550 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.695527 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:57.696036 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.696016 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-kube-api-access-4tbs9" (OuterVolumeSpecName: "kube-api-access-4tbs9") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "kube-api-access-4tbs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:25:57.702669 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.702638 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-web-config" (OuterVolumeSpecName: "web-config") pod "0b0370a1-7056-41a3-a854-a90c5fea7c2b" (UID: "0b0370a1-7056-41a3-a854-a90c5fea7c2b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:25:57.709446 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709424 2572 generic.go:356] "Generic (PLEG): container finished" podID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerID="195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" exitCode=0 Jul 06 12:25:57.709446 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709444 2572 generic.go:356] "Generic (PLEG): container finished" podID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerID="a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" exitCode=0 Jul 06 12:25:57.709553 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709454 2572 generic.go:356] "Generic (PLEG): container finished" podID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerID="1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" exitCode=0 Jul 06 12:25:57.709553 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709462 2572 generic.go:356] "Generic (PLEG): container finished" podID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerID="e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" exitCode=0 Jul 06 12:25:57.709553 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709469 2572 generic.go:356] "Generic (PLEG): container finished" podID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerID="e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" exitCode=0 Jul 06 12:25:57.709553 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709475 2572 generic.go:356] "Generic (PLEG): container finished" podID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerID="e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" exitCode=0 Jul 06 12:25:57.709553 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709504 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.709553 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709504 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerDied","Data":"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674"} Jul 06 12:25:57.709768 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709589 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerDied","Data":"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308"} Jul 06 12:25:57.709768 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerDied","Data":"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d"} Jul 06 12:25:57.709768 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709651 2572 scope.go:117] "RemoveContainer" containerID="195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" Jul 06 12:25:57.709768 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709690 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerDied","Data":"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408"} Jul 06 12:25:57.709768 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709734 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerDied","Data":"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484"} Jul 06 12:25:57.709929 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709774 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerDied","Data":"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d"} Jul 06 12:25:57.709929 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.709817 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0370a1-7056-41a3-a854-a90c5fea7c2b","Type":"ContainerDied","Data":"f1ba08a0e7c68ba284c6986076a8097c02190c2d4b3e0ad198657c0b8f5159c8"} Jul 06 12:25:57.717314 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.717065 2572 scope.go:117] "RemoveContainer" containerID="a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" Jul 06 12:25:57.725873 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.725850 2572 scope.go:117] "RemoveContainer" containerID="1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" Jul 06 12:25:57.732138 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.732111 2572 scope.go:117] "RemoveContainer" containerID="e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" Jul 06 12:25:57.739214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.739198 2572 scope.go:117] "RemoveContainer" containerID="e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" Jul 06 12:25:57.739556 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.739538 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 12:25:57.746488 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.746473 2572 scope.go:117] "RemoveContainer" containerID="e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" Jul 06 12:25:57.746904 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.746888 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 12:25:57.753057 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.753032 2572 scope.go:117] "RemoveContainer" containerID="cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229" Jul 06 12:25:57.759008 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.758993 2572 scope.go:117] "RemoveContainer" containerID="195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" Jul 06 12:25:57.759252 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:57.759232 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": container with ID starting with 195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674 not found: ID does not exist" containerID="195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" Jul 06 12:25:57.759303 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.759259 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674"} err="failed to get container status \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": rpc error: code = NotFound desc = could not find container \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": container with ID starting with 195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674 not found: ID does not exist" Jul 06 12:25:57.759303 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.759276 2572 scope.go:117] "RemoveContainer" containerID="a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" Jul 06 12:25:57.759519 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:57.759503 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": container with ID starting with a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308 not found: ID does not exist" containerID="a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" Jul 06 12:25:57.759594 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.759527 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308"} err="failed to get container status \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": rpc error: code = NotFound desc = could not find container \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": container with ID starting with a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308 not found: ID does not exist" Jul 06 12:25:57.759594 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.759549 2572 scope.go:117] "RemoveContainer" containerID="1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" Jul 06 12:25:57.759818 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:57.759803 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": container with ID starting with 1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d not found: ID does not exist" containerID="1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" Jul 06 12:25:57.759865 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.759823 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d"} err="failed to get container status \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": rpc error: code = NotFound desc = could not find container \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": container with ID starting with 1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d not found: ID does not exist" Jul 06 12:25:57.759865 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.759837 2572 scope.go:117] "RemoveContainer" containerID="e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" Jul 06 12:25:57.760056 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:57.760039 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": container with ID starting with e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408 not found: ID does not exist" containerID="e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" Jul 06 12:25:57.760112 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.760065 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408"} err="failed to get container status \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": rpc error: code = NotFound desc = could not find container \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": container with ID starting with e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408 not found: ID does not exist" Jul 06 12:25:57.760112 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.760085 2572 scope.go:117] "RemoveContainer" containerID="e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" Jul 06 12:25:57.760304 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:57.760288 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": container with ID starting with e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484 not found: ID does not exist" containerID="e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" Jul 06 12:25:57.760340 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.760309 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484"} err="failed to get container status \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": rpc error: code = NotFound desc = could not find container \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": container with ID starting with e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484 not found: ID does not exist" Jul 06 12:25:57.760340 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.760328 2572 scope.go:117] "RemoveContainer" containerID="e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" Jul 06 12:25:57.760520 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:57.760504 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": container with ID starting with e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d not found: ID does not exist" containerID="e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" Jul 06 12:25:57.760575 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.760526 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d"} err="failed to get container status \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": rpc error: code = NotFound desc = could not find container \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": container with ID starting with e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d not found: ID does not exist" Jul 06 12:25:57.760575 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.760544 2572 scope.go:117] "RemoveContainer" containerID="cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229" Jul 06 12:25:57.760762 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:25:57.760746 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": container with ID starting with cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229 not found: ID does not exist" containerID="cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229" Jul 06 12:25:57.760813 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.760766 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229"} err="failed to get container status \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": rpc error: code = NotFound desc = could not find container \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": container with ID starting with cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229 not found: ID does not exist" Jul 06 12:25:57.760813 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.760779 2572 scope.go:117] "RemoveContainer" containerID="195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" Jul 06 12:25:57.761001 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.760984 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674"} err="failed to get container status \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": rpc error: code = NotFound desc = could not find container \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": container with ID starting with 195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674 not found: ID does not exist" Jul 06 12:25:57.761047 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761002 2572 scope.go:117] "RemoveContainer" containerID="a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" Jul 06 12:25:57.761204 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761189 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308"} err="failed to get container status \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": rpc error: code = NotFound desc = could not find container \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": container with ID starting with a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308 not found: ID does not exist" Jul 06 12:25:57.761204 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761203 2572 scope.go:117] "RemoveContainer" containerID="1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" Jul 06 12:25:57.761400 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761385 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d"} err="failed to get container status \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": rpc error: code = NotFound desc = could not find container \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": container with ID starting with 1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d not found: ID does not exist" Jul 06 12:25:57.761445 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761401 2572 scope.go:117] "RemoveContainer" containerID="e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" Jul 06 12:25:57.761584 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761569 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408"} err="failed to get container status \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": rpc error: code = NotFound desc = could not find container \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": container with ID starting with e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408 not found: ID does not exist" Jul 06 12:25:57.761624 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761584 2572 scope.go:117] "RemoveContainer" containerID="e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" Jul 06 12:25:57.761798 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761781 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484"} err="failed to get container status \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": rpc error: code = NotFound desc = could not find container \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": container with ID starting with e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484 not found: ID does not exist" Jul 06 12:25:57.761852 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761800 2572 scope.go:117] "RemoveContainer" containerID="e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" Jul 06 12:25:57.762004 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.761988 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d"} err="failed to get container status \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": rpc error: code = NotFound desc = could not find container \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": container with ID starting with e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d not found: ID does not exist" Jul 06 12:25:57.762039 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762004 2572 scope.go:117] "RemoveContainer" containerID="cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229" Jul 06 12:25:57.762192 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762177 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229"} err="failed to get container status \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": rpc error: code = NotFound desc = could not find container \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": container with ID starting with cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229 not found: ID does not exist" Jul 06 12:25:57.762237 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762192 2572 scope.go:117] "RemoveContainer" containerID="195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" Jul 06 12:25:57.762377 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762362 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674"} err="failed to get container status \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": rpc error: code = NotFound desc = could not find container \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": container with ID starting with 195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674 not found: ID does not exist" Jul 06 12:25:57.762419 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762377 2572 scope.go:117] "RemoveContainer" containerID="a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" Jul 06 12:25:57.762607 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762591 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308"} err="failed to get container status \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": rpc error: code = NotFound desc = could not find container \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": container with ID starting with a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308 not found: ID does not exist" Jul 06 12:25:57.762607 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762606 2572 scope.go:117] "RemoveContainer" containerID="1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" Jul 06 12:25:57.762812 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762796 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d"} err="failed to get container status \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": rpc error: code = NotFound desc = could not find container \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": container with ID starting with 1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d not found: ID does not exist" Jul 06 12:25:57.762855 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762813 2572 scope.go:117] "RemoveContainer" containerID="e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" Jul 06 12:25:57.762988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762967 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408"} err="failed to get container status \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": rpc error: code = NotFound desc = could not find container \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": container with ID starting with e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408 not found: ID does not exist" Jul 06 12:25:57.763030 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.762992 2572 scope.go:117] "RemoveContainer" containerID="e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" Jul 06 12:25:57.763184 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.763168 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484"} err="failed to get container status \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": rpc error: code = NotFound desc = could not find container \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": container with ID starting with e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484 not found: ID does not exist" Jul 06 12:25:57.763221 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.763184 2572 scope.go:117] "RemoveContainer" containerID="e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" Jul 06 12:25:57.763393 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.763378 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d"} err="failed to get container status \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": rpc error: code = NotFound desc = could not find container \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": container with ID starting with e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d not found: ID does not exist" Jul 06 12:25:57.763431 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.763393 2572 scope.go:117] "RemoveContainer" containerID="cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229" Jul 06 12:25:57.763588 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.763564 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229"} err="failed to get container status \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": rpc error: code = NotFound desc = could not find container \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": container with ID starting with cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229 not found: ID does not exist" Jul 06 12:25:57.763631 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.763589 2572 scope.go:117] "RemoveContainer" containerID="195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" Jul 06 12:25:57.763834 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.763804 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674"} err="failed to get container status \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": rpc error: code = NotFound desc = could not find container \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": container with ID starting with 195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674 not found: ID does not exist" Jul 06 12:25:57.763887 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.763837 2572 scope.go:117] "RemoveContainer" containerID="a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" Jul 06 12:25:57.764037 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764019 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308"} err="failed to get container status \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": rpc error: code = NotFound desc = could not find container \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": container with ID starting with a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308 not found: ID does not exist" Jul 06 12:25:57.764108 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764041 2572 scope.go:117] "RemoveContainer" containerID="1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" Jul 06 12:25:57.764245 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764229 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d"} err="failed to get container status \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": rpc error: code = NotFound desc = could not find container \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": container with ID starting with 1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d not found: ID does not exist" Jul 06 12:25:57.764289 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764245 2572 scope.go:117] "RemoveContainer" containerID="e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" Jul 06 12:25:57.764445 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764430 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408"} err="failed to get container status \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": rpc error: code = NotFound desc = could not find container \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": container with ID starting with e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408 not found: ID does not exist" Jul 06 12:25:57.764488 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764444 2572 scope.go:117] "RemoveContainer" containerID="e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" Jul 06 12:25:57.764626 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764611 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484"} err="failed to get container status \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": rpc error: code = NotFound desc = could not find container \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": container with ID starting with e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484 not found: ID does not exist" Jul 06 12:25:57.764691 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764626 2572 scope.go:117] "RemoveContainer" containerID="e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" Jul 06 12:25:57.764822 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764808 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d"} err="failed to get container status \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": rpc error: code = NotFound desc = could not find container \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": container with ID starting with e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d not found: ID does not exist" Jul 06 12:25:57.764868 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764822 2572 scope.go:117] "RemoveContainer" containerID="cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229" Jul 06 12:25:57.765013 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.764996 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229"} err="failed to get container status \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": rpc error: code = NotFound desc = could not find container \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": container with ID starting with cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229 not found: ID does not exist" Jul 06 12:25:57.765140 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.765015 2572 scope.go:117] "RemoveContainer" containerID="195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" Jul 06 12:25:57.765250 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.765229 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674"} err="failed to get container status \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": rpc error: code = NotFound desc = could not find container \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": container with ID starting with 195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674 not found: ID does not exist" Jul 06 12:25:57.765298 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.765252 2572 scope.go:117] "RemoveContainer" containerID="a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" Jul 06 12:25:57.765462 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.765446 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308"} err="failed to get container status \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": rpc error: code = NotFound desc = could not find container \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": container with ID starting with a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308 not found: ID does not exist" Jul 06 12:25:57.765505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.765462 2572 scope.go:117] "RemoveContainer" containerID="1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" Jul 06 12:25:57.765677 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.765642 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d"} err="failed to get container status \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": rpc error: code = NotFound desc = could not find container \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": container with ID starting with 1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d not found: ID does not exist" Jul 06 12:25:57.765677 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.765674 2572 scope.go:117] "RemoveContainer" containerID="e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" Jul 06 12:25:57.765864 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.765849 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408"} err="failed to get container status \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": rpc error: code = NotFound desc = could not find container \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": container with ID starting with e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408 not found: ID does not exist" Jul 06 12:25:57.765864 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.765864 2572 scope.go:117] "RemoveContainer" containerID="e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" Jul 06 12:25:57.766080 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.766065 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484"} err="failed to get container status \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": rpc error: code = NotFound desc = could not find container \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": container with ID starting with e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484 not found: ID does not exist" Jul 06 12:25:57.766142 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.766081 2572 scope.go:117] "RemoveContainer" containerID="e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" Jul 06 12:25:57.766294 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.766275 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d"} err="failed to get container status \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": rpc error: code = NotFound desc = could not find container \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": container with ID starting with e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d not found: ID does not exist" Jul 06 12:25:57.766336 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.766295 2572 scope.go:117] "RemoveContainer" containerID="cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229" Jul 06 12:25:57.766506 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.766486 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229"} err="failed to get container status \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": rpc error: code = NotFound desc = could not find container \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": container with ID starting with cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229 not found: ID does not exist" Jul 06 12:25:57.766543 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.766509 2572 scope.go:117] "RemoveContainer" containerID="195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674" Jul 06 12:25:57.766790 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.766773 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674"} err="failed to get container status \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": rpc error: code = NotFound desc = could not find container \"195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674\": container with ID starting with 195302dd7489a32ac5c4e4026528aed7a1430ca0d70f570749511e9399e3c674 not found: ID does not exist" Jul 06 12:25:57.766851 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.766789 2572 scope.go:117] "RemoveContainer" containerID="a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308" Jul 06 12:25:57.767002 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.766984 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308"} err="failed to get container status \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": rpc error: code = NotFound desc = could not find container \"a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308\": container with ID starting with a727fc4b02345f3e63bd6ee5b86ea0ed5e177c6f25471f973f6093182e490308 not found: ID does not exist" Jul 06 12:25:57.767056 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.767004 2572 scope.go:117] "RemoveContainer" containerID="1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d" Jul 06 12:25:57.767206 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.767191 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d"} err="failed to get container status \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": rpc error: code = NotFound desc = could not find container \"1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d\": container with ID starting with 1513b9d766e3a851249ceebca57c24bfa4577064150e793447662d0a5df8225d not found: ID does not exist" Jul 06 12:25:57.767250 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.767207 2572 scope.go:117] "RemoveContainer" containerID="e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408" Jul 06 12:25:57.767395 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.767378 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408"} err="failed to get container status \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": rpc error: code = NotFound desc = could not find container \"e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408\": container with ID starting with e6237029a57f8ed3a2d30205de496fbe7232652459c67ea01a415ff875ccf408 not found: ID does not exist" Jul 06 12:25:57.767439 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.767397 2572 scope.go:117] "RemoveContainer" containerID="e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484" Jul 06 12:25:57.767620 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.767603 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484"} err="failed to get container status \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": rpc error: code = NotFound desc = could not find container \"e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484\": container with ID starting with e35e3b071762677a6fc4cd9a472e34741e1c065d98fe03d9ae769c4f881b0484 not found: ID does not exist" Jul 06 12:25:57.767687 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.767620 2572 scope.go:117] "RemoveContainer" containerID="e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d" Jul 06 12:25:57.767860 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.767842 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d"} err="failed to get container status \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": rpc error: code = NotFound desc = could not find container \"e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d\": container with ID starting with e980497bc78df9098a4fef281bd54a045aa010180e0387f65d09ee719a45a59d not found: ID does not exist" Jul 06 12:25:57.767909 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.767862 2572 scope.go:117] "RemoveContainer" containerID="cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229" Jul 06 12:25:57.768082 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.768065 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229"} err="failed to get container status \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": rpc error: code = NotFound desc = could not find container \"cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229\": container with ID starting with cc0e0f4689ce50664d70b29fdb04fcf65673b8a64921126c7487674dfa99e229 not found: ID does not exist" Jul 06 12:25:57.785713 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.785693 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 12:25:57.785955 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.785944 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="config-reloader" Jul 06 12:25:57.785999 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.785957 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="config-reloader" Jul 06 12:25:57.785999 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.785965 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy-web" Jul 06 12:25:57.785999 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.785972 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy-web" Jul 06 12:25:57.785999 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.785981 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy-web" Jul 06 12:25:57.785999 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.785986 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy-web" Jul 06 12:25:57.785999 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.785992 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="init-config-reloader" Jul 06 12:25:57.785999 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.785997 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="init-config-reloader" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786006 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="init-config-reloader" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786012 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="init-config-reloader" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786018 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="prometheus" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786023 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="prometheus" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786031 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy-thanos" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786036 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy-thanos" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786042 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="thanos-sidecar" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786047 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="thanos-sidecar" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786053 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786058 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786067 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="prom-label-proxy" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786072 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="prom-label-proxy" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786079 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="alertmanager" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786084 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="alertmanager" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786090 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="config-reloader" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786095 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="config-reloader" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786100 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786104 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786109 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy-metric" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786114 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy-metric" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786151 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="config-reloader" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786157 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="prometheus" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786162 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="config-reloader" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786167 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786174 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy-thanos" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786180 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="thanos-sidecar" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786185 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" containerName="kube-rbac-proxy-web" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786191 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy-web" Jul 06 12:25:57.786186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786196 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="alertmanager" Jul 06 12:25:57.786961 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786201 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy-metric" Jul 06 12:25:57.786961 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786206 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="prom-label-proxy" Jul 06 12:25:57.786961 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.786211 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b32201d-892d-40dd-8f5e-fe8962e25a73" containerName="kube-rbac-proxy" Jul 06 12:25:57.791170 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791154 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.791647 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791622 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config-out\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791647 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791645 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791670 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-metrics-client-ca\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791682 2572 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-kube-rbac-proxy\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791691 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-db\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791699 2572 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-metrics-client-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791724 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-web-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791733 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tbs9\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-kube-api-access-4tbs9\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791741 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791750 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0370a1-7056-41a3-a854-a90c5fea7c2b-tls-assets\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791758 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791765 2572 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-grpc-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791777 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791785 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.791787 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791793 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0370a1-7056-41a3-a854-a90c5fea7c2b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.792152 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791802 2572 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.792152 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.791810 2572 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0370a1-7056-41a3-a854-a90c5fea7c2b-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:25:57.796319 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.796291 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 12:25:57.796576 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.796559 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 12:25:57.796800 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.796786 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 12:25:57.796872 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.796821 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 12:25:57.797443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.797406 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 12:25:57.797541 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.797406 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 12:25:57.797724 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.797693 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 12:25:57.797816 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.797734 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 12:25:57.798438 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.798419 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 12:25:57.798530 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.798419 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1gt5i394uragt\"" Jul 06 12:25:57.798530 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.798420 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 12:25:57.801560 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.801543 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 12:25:57.802757 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.802743 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 12:25:57.802975 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.802962 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 12:25:57.803224 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.803208 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l76dc\"" Jul 06 12:25:57.803499 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.803485 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 12:25:57.807056 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.807041 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 12:25:57.813462 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.813444 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 12:25:57.830546 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.830522 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0370a1-7056-41a3-a854-a90c5fea7c2b" path="/var/lib/kubelet/pods/0b0370a1-7056-41a3-a854-a90c5fea7c2b/volumes" Jul 06 12:25:57.892270 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892270 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892242 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892270 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892262 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892439 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892285 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892439 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892328 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892439 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892385 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892439 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892439 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892608 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892467 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892608 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892500 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e9e2467-f132-46c3-b7ee-75ec491effd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892608 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892527 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892608 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892565 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892608 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892817 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892617 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qcg\" (UniqueName: \"kubernetes.io/projected/0e9e2467-f132-46c3-b7ee-75ec491effd8-kube-api-access-79qcg\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892817 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892640 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892817 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892686 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892817 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892710 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-config\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892817 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892817 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892750 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.892817 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.892766 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e9e2467-f132-46c3-b7ee-75ec491effd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994022 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.993997 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994022 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994084 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994118 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994148 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e9e2467-f132-46c3-b7ee-75ec491effd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994166 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994187 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994218 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qcg\" (UniqueName: \"kubernetes.io/projected/0e9e2467-f132-46c3-b7ee-75ec491effd8-kube-api-access-79qcg\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994289 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994310 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-config\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994336 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994363 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e9e2467-f132-46c3-b7ee-75ec491effd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994456 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.994517 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.994497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.995067 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.995037 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.995067 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.995055 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.995391 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.995362 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997099 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997226 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997250 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997267 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997595 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997383 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997791 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997764 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997884 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997790 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997884 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997867 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997978 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997899 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e9e2467-f132-46c3-b7ee-75ec491effd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.997978 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.997926 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.998355 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.998332 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-config\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.998816 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.998796 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.999840 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.999822 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e9e2467-f132-46c3-b7ee-75ec491effd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:57.999915 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:57.999866 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:58.000270 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:58.000250 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e9e2467-f132-46c3-b7ee-75ec491effd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:58.001009 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:58.000993 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e9e2467-f132-46c3-b7ee-75ec491effd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:58.007038 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:58.007020 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qcg\" (UniqueName: \"kubernetes.io/projected/0e9e2467-f132-46c3-b7ee-75ec491effd8-kube-api-access-79qcg\") pod \"prometheus-k8s-0\" (UID: \"0e9e2467-f132-46c3-b7ee-75ec491effd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:58.099791 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:58.099761 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:25:58.244400 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:25:58.244348 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9e2467_f132_46c3_b7ee_75ec491effd8.slice/crio-3b21bff888006485c225fea6d18f8957884c6e2930426cc62e30682af382cdf0 WatchSource:0}: Error finding container 3b21bff888006485c225fea6d18f8957884c6e2930426cc62e30682af382cdf0: Status 404 returned error can't find the container with id 3b21bff888006485c225fea6d18f8957884c6e2930426cc62e30682af382cdf0 Jul 06 12:25:58.244765 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:58.244742 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 12:25:58.715007 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:58.714973 2572 generic.go:356] "Generic (PLEG): container finished" podID="0e9e2467-f132-46c3-b7ee-75ec491effd8" containerID="11939812eb5e83011bc567e1e7ffa85d95ad692772d3cfdc75f8ce7d5e771fb7" exitCode=0 Jul 06 12:25:58.715425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:58.715006 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e9e2467-f132-46c3-b7ee-75ec491effd8","Type":"ContainerDied","Data":"11939812eb5e83011bc567e1e7ffa85d95ad692772d3cfdc75f8ce7d5e771fb7"} Jul 06 12:25:58.715425 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:58.715084 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e9e2467-f132-46c3-b7ee-75ec491effd8","Type":"ContainerStarted","Data":"3b21bff888006485c225fea6d18f8957884c6e2930426cc62e30682af382cdf0"} Jul 06 12:25:59.723113 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:59.723072 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e9e2467-f132-46c3-b7ee-75ec491effd8","Type":"ContainerStarted","Data":"8c56b7ec28a36a00e399eefd29838e352ba88ba7c72b79124bd2213adebb96e8"} Jul 06 12:25:59.723526 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:59.723177 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e9e2467-f132-46c3-b7ee-75ec491effd8","Type":"ContainerStarted","Data":"bd3456117954b30171adc5c02b05d5ed154483a6b9c99f40302aeb780c220e09"} Jul 06 12:25:59.723526 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:59.723206 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e9e2467-f132-46c3-b7ee-75ec491effd8","Type":"ContainerStarted","Data":"11dca19b1062662a4ec3ccb9f0548ee0d0747e676bc27c76c95521d2a541b15d"} Jul 06 12:25:59.723526 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:59.723234 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e9e2467-f132-46c3-b7ee-75ec491effd8","Type":"ContainerStarted","Data":"ed31b23a14cf06606ffff35bb673d15fba1efe505ff002d6c089f4b7282fc17f"} Jul 06 12:25:59.723526 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:59.723264 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e9e2467-f132-46c3-b7ee-75ec491effd8","Type":"ContainerStarted","Data":"3f4d416686d9a278bc56daf703c1ff0ff8853b1b8bca6c223e37392daee300bf"} Jul 06 12:25:59.723526 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:59.723289 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e9e2467-f132-46c3-b7ee-75ec491effd8","Type":"ContainerStarted","Data":"5f4a690f2b108c43a6a3201ce1ee1957edcf391740a006b7cf4d42d4e22fac9a"} Jul 06 12:25:59.761333 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:25:59.761286 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7420513619999998 podStartE2EDuration="2.761272301s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.220939ms" isStatefulPod=false podCreationTimestamp="2026-07-06 12:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:25:59.758548024 +0000 UTC m=+175.837096053" watchObservedRunningTime="2026-07-06 12:25:59.761272301 +0000 UTC m=+175.839820367" Jul 06 12:26:00.576993 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:00.576959 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:00.643243 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:00.643213 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:00.887244 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:00.887155 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:00.995994 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:00.995960 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:01.060729 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:01.060697 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:03.100763 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:03.100729 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:26:04.831902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:04.831864 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:09.638921 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:09.638889 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:09.738438 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:09.738416 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:38.609434 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:38.609394 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:38.683652 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:38.683617 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:44.673701 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:44.673647 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:44.770019 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:44.769986 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:26:58.100124 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:58.100079 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:26:58.115302 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:58.115277 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:26:58.948606 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:26:58.948584 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 12:27:11.843973 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:11.843941 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:12.185579 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:12.185474 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:15.806837 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:15.806808 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:16.022958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:16.022923 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:16.785237 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:16.785204 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:16.900859 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:16.900826 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:17.920140 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:17.920109 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:18.066841 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:18.066811 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:20.184363 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:20.184332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:20.405601 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:20.405564 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:22.039909 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:22.039879 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:22.098212 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:22.098177 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:24.701455 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:24.701419 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:26.957911 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:26.957835 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:27.039772 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:27.039741 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:28.016093 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:28.016059 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:29.168525 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:29.168496 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:29.274800 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:29.274765 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:34.213939 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:34.213906 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:34.308353 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:34.308320 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:36.364455 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:36.364424 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:51.309981 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:51.309944 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:51.401421 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:51.401395 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:51.891350 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:51.891314 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:52.341298 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.341266 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:52.451591 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.451550 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:52.508616 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.508587 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9"] Jul 06 12:27:52.512060 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.512039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.514790 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.514771 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 12:27:52.514965 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.514950 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 12:27:52.515070 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.515057 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 12:27:52.515070 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.515060 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 12:27:52.515306 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.515292 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kssh9\"" Jul 06 12:27:52.524171 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.524152 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9"] Jul 06 12:27:52.658289 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.658215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl9vz\" (UniqueName: \"kubernetes.io/projected/b864c88a-0dce-4d8b-8a55-0259b4617bdd-kube-api-access-tl9vz\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.658289 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.658265 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b864c88a-0dce-4d8b-8a55-0259b4617bdd-tmp\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.658289 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.658288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b864c88a-0dce-4d8b-8a55-0259b4617bdd-manifests\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.658536 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.658305 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b864c88a-0dce-4d8b-8a55-0259b4617bdd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.658536 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.658364 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b864c88a-0dce-4d8b-8a55-0259b4617bdd-webhook-cert\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.687990 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.687957 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:52.759145 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.759107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b864c88a-0dce-4d8b-8a55-0259b4617bdd-webhook-cert\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.759293 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.759181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl9vz\" (UniqueName: \"kubernetes.io/projected/b864c88a-0dce-4d8b-8a55-0259b4617bdd-kube-api-access-tl9vz\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.759293 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.759203 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b864c88a-0dce-4d8b-8a55-0259b4617bdd-tmp\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.759293 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.759223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b864c88a-0dce-4d8b-8a55-0259b4617bdd-manifests\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.759293 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.759239 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b864c88a-0dce-4d8b-8a55-0259b4617bdd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.759768 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.759740 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b864c88a-0dce-4d8b-8a55-0259b4617bdd-manifests\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.761471 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.761454 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b864c88a-0dce-4d8b-8a55-0259b4617bdd-tmp\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.761746 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.761723 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b864c88a-0dce-4d8b-8a55-0259b4617bdd-webhook-cert\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.761806 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.761792 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b864c88a-0dce-4d8b-8a55-0259b4617bdd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.776812 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.776786 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl9vz\" (UniqueName: \"kubernetes.io/projected/b864c88a-0dce-4d8b-8a55-0259b4617bdd-kube-api-access-tl9vz\") pod \"opendatahub-operator-controller-manager-67cf9968b4-dvbh9\" (UID: \"b864c88a-0dce-4d8b-8a55-0259b4617bdd\") " pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.803247 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.803210 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:52.822198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.822178 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:52.903507 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.903429 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:52.954531 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.954510 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9"] Jul 06 12:27:52.956609 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:27:52.956588 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb864c88a_0dce_4d8b_8a55_0259b4617bdd.slice/crio-10b5c3844a6e3a34cf251ff1b4410b8986a9a25de08b37313857bdfefe9ecc3c WatchSource:0}: Error finding container 10b5c3844a6e3a34cf251ff1b4410b8986a9a25de08b37313857bdfefe9ecc3c: Status 404 returned error can't find the container with id 10b5c3844a6e3a34cf251ff1b4410b8986a9a25de08b37313857bdfefe9ecc3c Jul 06 12:27:52.999708 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:52.999680 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:53.116424 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:53.116393 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:53.119538 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:53.119515 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" event={"ID":"b864c88a-0dce-4d8b-8a55-0259b4617bdd","Type":"ContainerStarted","Data":"10b5c3844a6e3a34cf251ff1b4410b8986a9a25de08b37313857bdfefe9ecc3c"} Jul 06 12:27:53.271107 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:53.271077 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:56.135521 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:56.135493 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" event={"ID":"b864c88a-0dce-4d8b-8a55-0259b4617bdd","Type":"ContainerStarted","Data":"0f050e71ad61034a6331ea172c5732f03e30700f1f471f372f828bf32196951c"} Jul 06 12:27:57.988089 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:57.988056 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:58.144723 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:58.144686 2572 generic.go:356] "Generic (PLEG): container finished" podID="b864c88a-0dce-4d8b-8a55-0259b4617bdd" containerID="0f050e71ad61034a6331ea172c5732f03e30700f1f471f372f828bf32196951c" exitCode=0 Jul 06 12:27:58.144877 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:58.144735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" event={"ID":"b864c88a-0dce-4d8b-8a55-0259b4617bdd","Type":"ContainerDied","Data":"0f050e71ad61034a6331ea172c5732f03e30700f1f471f372f828bf32196951c"} Jul 06 12:27:58.186877 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:58.186847 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:27:59.150406 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:59.150367 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" event={"ID":"b864c88a-0dce-4d8b-8a55-0259b4617bdd","Type":"ContainerStarted","Data":"a2a451ff943cb6e3e6bbda4a1543590efebc32960fcda7ec3d0d2c68e8a23e98"} Jul 06 12:27:59.150808 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:59.150490 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:27:59.208602 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:27:59.208552 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" podStartSLOduration=2.625604257 podStartE2EDuration="7.208535772s" totalImagesPullingTime="3.488190399s" totalInitContainerRuntime="1.094741116s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:27:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:27:59.202909967 +0000 UTC m=+295.281457994" watchObservedRunningTime="2026-07-06 12:27:59.208535772 +0000 UTC m=+295.287083798" Jul 06 12:28:03.639170 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:03.639139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:05.717589 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:05.717552 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 12:28:10.156133 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:10.156097 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67cf9968b4-dvbh9" Jul 06 12:28:12.050198 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:12.050164 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:13.259941 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:13.259910 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:13.361383 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:13.361358 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:13.461435 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:13.461407 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:13.710098 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:13.710069 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:19.209205 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.209166 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q"] Jul 06 12:28:19.211303 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.211287 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.215314 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.215287 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 12:28:19.216629 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.216610 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zzthp\"" Jul 06 12:28:19.216940 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.216921 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 12:28:19.217018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.216934 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 12:28:19.217018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.216921 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 12:28:19.217123 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.217022 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 12:28:19.228116 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.228095 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q"] Jul 06 12:28:19.273926 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.273895 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw8g\" (UniqueName: \"kubernetes.io/projected/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-kube-api-access-lqw8g\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.274036 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.273942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-manager-config\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.274036 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.273998 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-cert\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.274123 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.274066 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-metrics-cert\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.375363 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.375320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw8g\" (UniqueName: \"kubernetes.io/projected/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-kube-api-access-lqw8g\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.375363 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.375373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-manager-config\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.375618 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.375404 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-cert\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.375618 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.375446 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-metrics-cert\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.376065 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.376033 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-manager-config\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.378030 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.378005 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-metrics-cert\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.378123 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.378013 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-cert\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.385502 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.385479 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw8g\" (UniqueName: \"kubernetes.io/projected/5634c8f1-5d2c-46ae-adf1-c0389b6950ce-kube-api-access-lqw8g\") pod \"lws-controller-manager-69bb7bb478-knv5q\" (UID: \"5634c8f1-5d2c-46ae-adf1-c0389b6950ce\") " pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.520336 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.520257 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:19.659884 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.659854 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q"] Jul 06 12:28:19.662314 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:28:19.662286 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5634c8f1_5d2c_46ae_adf1_c0389b6950ce.slice/crio-8d9af5faa7da718d2c8e8a21197e5852bea50d9105f968d9c6198f81b9ef5a19 WatchSource:0}: Error finding container 8d9af5faa7da718d2c8e8a21197e5852bea50d9105f968d9c6198f81b9ef5a19: Status 404 returned error can't find the container with id 8d9af5faa7da718d2c8e8a21197e5852bea50d9105f968d9c6198f81b9ef5a19 Jul 06 12:28:19.664225 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:19.664211 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 12:28:20.225926 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:20.225889 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" event={"ID":"5634c8f1-5d2c-46ae-adf1-c0389b6950ce","Type":"ContainerStarted","Data":"8d9af5faa7da718d2c8e8a21197e5852bea50d9105f968d9c6198f81b9ef5a19"} Jul 06 12:28:22.235780 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:22.235739 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" event={"ID":"5634c8f1-5d2c-46ae-adf1-c0389b6950ce","Type":"ContainerStarted","Data":"48279bf40805a5944996d26111d9bcc0fce3e056e776f400399a2e41d6d34e5a"} Jul 06 12:28:22.236146 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:22.235857 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:22.875790 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:22.875760 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:22.999944 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:22.999917 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:23.399772 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.399741 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:23.430941 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.430884 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" podStartSLOduration=1.97378935 podStartE2EDuration="4.430864774s" totalImagesPullingTime="2.457075424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:28:22.279317027 +0000 UTC m=+318.357865055" watchObservedRunningTime="2026-07-06 12:28:23.430864774 +0000 UTC m=+319.509412802" Jul 06 12:28:23.431458 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.431436 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5"] Jul 06 12:28:23.438059 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.438038 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:23.441280 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.441246 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 12:28:23.441399 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.441245 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 12:28:23.443024 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.442799 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 12:28:23.443024 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.442857 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kz7bz\"" Jul 06 12:28:23.443024 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.442898 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 12:28:23.451513 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.451490 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5"] Jul 06 12:28:23.513722 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.513689 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccc16fc-9866-4c75-8559-6717f17ddde3-tls-certs\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:23.513722 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.513727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ccc16fc-9866-4c75-8559-6717f17ddde3-tmp\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:23.513926 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.513778 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8znml\" (UniqueName: \"kubernetes.io/projected/7ccc16fc-9866-4c75-8559-6717f17ddde3-kube-api-access-8znml\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:23.614222 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.614189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ccc16fc-9866-4c75-8559-6717f17ddde3-tmp\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:23.614391 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.614251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8znml\" (UniqueName: \"kubernetes.io/projected/7ccc16fc-9866-4c75-8559-6717f17ddde3-kube-api-access-8znml\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:23.614391 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.614295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccc16fc-9866-4c75-8559-6717f17ddde3-tls-certs\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:23.614480 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:28:23.614398 2572 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 06 12:28:23.614480 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:28:23.614474 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ccc16fc-9866-4c75-8559-6717f17ddde3-tls-certs podName:7ccc16fc-9866-4c75-8559-6717f17ddde3 nodeName:}" failed. No retries permitted until 2026-07-06 12:28:24.114455017 +0000 UTC m=+320.193003040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7ccc16fc-9866-4c75-8559-6717f17ddde3-tls-certs") pod "kube-auth-proxy-7d7ff6d984-kwvk5" (UID: "7ccc16fc-9866-4c75-8559-6717f17ddde3") : secret "kube-auth-proxy-tls" not found Jul 06 12:28:23.616469 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.616439 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ccc16fc-9866-4c75-8559-6717f17ddde3-tmp\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:23.638674 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.638629 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8znml\" (UniqueName: \"kubernetes.io/projected/7ccc16fc-9866-4c75-8559-6717f17ddde3-kube-api-access-8znml\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:23.761223 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:23.761189 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:24.120034 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:24.119947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccc16fc-9866-4c75-8559-6717f17ddde3-tls-certs\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:24.122397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:24.122377 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccc16fc-9866-4c75-8559-6717f17ddde3-tls-certs\") pod \"kube-auth-proxy-7d7ff6d984-kwvk5\" (UID: \"7ccc16fc-9866-4c75-8559-6717f17ddde3\") " pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:24.351931 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:24.351898 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" Jul 06 12:28:24.493641 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:24.493608 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:24.511370 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:24.511341 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5"] Jul 06 12:28:24.516472 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:28:24.516444 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccc16fc_9866_4c75_8559_6717f17ddde3.slice/crio-4ba3bd0321932bb7fef56b00f1cbc46c13bc5c6395d6d547e8c047ce10cc027a WatchSource:0}: Error finding container 4ba3bd0321932bb7fef56b00f1cbc46c13bc5c6395d6d547e8c047ce10cc027a: Status 404 returned error can't find the container with id 4ba3bd0321932bb7fef56b00f1cbc46c13bc5c6395d6d547e8c047ce10cc027a Jul 06 12:28:24.646252 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:24.646224 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:24.769612 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:24.769578 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:24.980878 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:24.980847 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:25.248426 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:25.248389 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" event={"ID":"7ccc16fc-9866-4c75-8559-6717f17ddde3","Type":"ContainerStarted","Data":"4ba3bd0321932bb7fef56b00f1cbc46c13bc5c6395d6d547e8c047ce10cc027a"} Jul 06 12:28:25.844971 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:25.844941 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:25.997073 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:25.997041 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:26.132766 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:26.132687 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:27.805113 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:27.805083 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:28.261332 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:28.261296 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" event={"ID":"7ccc16fc-9866-4c75-8559-6717f17ddde3","Type":"ContainerStarted","Data":"d91809021ee8d4ce61614c417073082174e2877c29b5d5ce0ac7dbc11cfd9211"} Jul 06 12:28:28.291488 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:28.291442 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d7ff6d984-kwvk5" podStartSLOduration=2.222257734 podStartE2EDuration="5.291426298s" totalImagesPullingTime="3.069168564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:28:28.290625018 +0000 UTC m=+324.369173045" watchObservedRunningTime="2026-07-06 12:28:28.291426298 +0000 UTC m=+324.369974378" Jul 06 12:28:32.361569 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:32.361521 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:32.593022 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:32.592989 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:33.241078 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:33.241047 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69bb7bb478-knv5q" Jul 06 12:28:33.643004 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:33.642925 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:33.899933 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:33.899860 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:28:34.151029 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:28:34.150944 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:13.660178 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:13.660105 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:13.864506 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:13.864473 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:14.110086 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:14.110054 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:42.635186 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:42.635150 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:43.244385 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:43.244357 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:43.692044 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:43.692017 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:43.794520 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:43.794491 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:43.895158 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:43.895060 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:44.453371 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:44.453332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:44.570790 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:44.570759 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:46.912966 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:46.912936 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:47.108589 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:47.108559 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:47.399309 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:47.399281 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:49.530046 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:49.530015 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:51.831584 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:51.831547 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:52.576596 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:52.576563 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:54.026960 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:54.026927 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:54.259431 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:54.259401 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:54.918106 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:54.918072 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:55.022772 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:55.022735 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:55.125702 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:55.125654 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:56.961499 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:56.961454 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:57.178003 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:57.177971 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:57.505996 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:57.505967 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:29:57.853866 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:29:57.853791 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:01.064739 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.064708 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w"] Jul 06 12:30:01.067732 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.067706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.068146 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.068110 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:01.070635 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.070615 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 12:30:01.071151 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.071130 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v82jm\"" Jul 06 12:30:01.071235 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.071130 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 12:30:01.071911 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.071891 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 12:30:01.072009 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.071891 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 12:30:01.081350 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.081326 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w"] Jul 06 12:30:01.087376 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.087354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/53fbd44b-b0d2-490e-9c16-c3b0de832a6f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sqv7w\" (UID: \"53fbd44b-b0d2-490e-9c16-c3b0de832a6f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.087480 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.087394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcbmv\" (UniqueName: \"kubernetes.io/projected/53fbd44b-b0d2-490e-9c16-c3b0de832a6f-kube-api-access-fcbmv\") pod \"kuadrant-console-plugin-66b5774985-sqv7w\" (UID: \"53fbd44b-b0d2-490e-9c16-c3b0de832a6f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.087480 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.087433 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53fbd44b-b0d2-490e-9c16-c3b0de832a6f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-sqv7w\" (UID: \"53fbd44b-b0d2-490e-9c16-c3b0de832a6f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.188349 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.188316 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53fbd44b-b0d2-490e-9c16-c3b0de832a6f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-sqv7w\" (UID: \"53fbd44b-b0d2-490e-9c16-c3b0de832a6f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.188516 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.188378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/53fbd44b-b0d2-490e-9c16-c3b0de832a6f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sqv7w\" (UID: \"53fbd44b-b0d2-490e-9c16-c3b0de832a6f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.188516 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.188406 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcbmv\" (UniqueName: \"kubernetes.io/projected/53fbd44b-b0d2-490e-9c16-c3b0de832a6f-kube-api-access-fcbmv\") pod \"kuadrant-console-plugin-66b5774985-sqv7w\" (UID: \"53fbd44b-b0d2-490e-9c16-c3b0de832a6f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.189053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.189030 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53fbd44b-b0d2-490e-9c16-c3b0de832a6f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-sqv7w\" (UID: \"53fbd44b-b0d2-490e-9c16-c3b0de832a6f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.190740 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.190719 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/53fbd44b-b0d2-490e-9c16-c3b0de832a6f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sqv7w\" (UID: \"53fbd44b-b0d2-490e-9c16-c3b0de832a6f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.210313 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.210288 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcbmv\" (UniqueName: \"kubernetes.io/projected/53fbd44b-b0d2-490e-9c16-c3b0de832a6f-kube-api-access-fcbmv\") pod \"kuadrant-console-plugin-66b5774985-sqv7w\" (UID: \"53fbd44b-b0d2-490e-9c16-c3b0de832a6f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.377253 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.377143 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" Jul 06 12:30:01.444306 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.444264 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:01.515650 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.515616 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w"] Jul 06 12:30:01.519369 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:30:01.519346 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53fbd44b_b0d2_490e_9c16_c3b0de832a6f.slice/crio-f47efae23e24c13623c0cf0ea57debe32b92dc1356394fe5d28b2ef3c9aa5816 WatchSource:0}: Error finding container f47efae23e24c13623c0cf0ea57debe32b92dc1356394fe5d28b2ef3c9aa5816: Status 404 returned error can't find the container with id f47efae23e24c13623c0cf0ea57debe32b92dc1356394fe5d28b2ef3c9aa5816 Jul 06 12:30:01.602087 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:01.602058 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" event={"ID":"53fbd44b-b0d2-490e-9c16-c3b0de832a6f","Type":"ContainerStarted","Data":"f47efae23e24c13623c0cf0ea57debe32b92dc1356394fe5d28b2ef3c9aa5816"} Jul 06 12:30:02.395382 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:02.395350 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:02.818148 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:02.818108 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:09.140430 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:09.140397 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:09.391991 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:09.391915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:13.534395 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:13.534356 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:13.630046 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:13.630008 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:17.543528 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:17.543489 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:17.649040 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:17.649000 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:18.100222 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:18.100193 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:18.233972 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:18.233924 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:22.407120 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:22.407076 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:22.526995 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:22.526963 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:24.709634 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:24.709599 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" event={"ID":"53fbd44b-b0d2-490e-9c16-c3b0de832a6f","Type":"ContainerStarted","Data":"33aa9d3647ecb6e47fbfe3d4e46aebb0c8518beb423bd7a9e6987cd72d3b35c9"} Jul 06 12:30:24.763270 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:24.763226 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sqv7w" podStartSLOduration=1.116106438 podStartE2EDuration="23.763211569s" totalImagesPullingTime="22.647105131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:30:24.754271501 +0000 UTC m=+440.832819528" watchObservedRunningTime="2026-07-06 12:30:24.763211569 +0000 UTC m=+440.841759595" Jul 06 12:30:28.697230 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:28.697158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:28.864042 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:28.864010 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:33.344834 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:33.344802 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:33.485005 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:33.484945 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:44.612070 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:44.612036 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:44.728029 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:44.728000 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:47.113561 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.113527 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:47.320386 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.320352 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:47.658653 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.658620 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:30:47.690960 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.690918 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:30:47.691126 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.691046 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:30:47.694302 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.694281 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 12:30:47.778785 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.778752 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d43bff14-4240-430d-972d-605f24dcf10b-config-file\") pod \"limitador-limitador-54d6d988d5-ksj9v\" (UID: \"d43bff14-4240-430d-972d-605f24dcf10b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:30:47.778785 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.778786 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh698\" (UniqueName: \"kubernetes.io/projected/d43bff14-4240-430d-972d-605f24dcf10b-kube-api-access-lh698\") pod \"limitador-limitador-54d6d988d5-ksj9v\" (UID: \"d43bff14-4240-430d-972d-605f24dcf10b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:30:47.879484 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.879451 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d43bff14-4240-430d-972d-605f24dcf10b-config-file\") pod \"limitador-limitador-54d6d988d5-ksj9v\" (UID: \"d43bff14-4240-430d-972d-605f24dcf10b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:30:47.879484 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.879492 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh698\" (UniqueName: \"kubernetes.io/projected/d43bff14-4240-430d-972d-605f24dcf10b-kube-api-access-lh698\") pod \"limitador-limitador-54d6d988d5-ksj9v\" (UID: \"d43bff14-4240-430d-972d-605f24dcf10b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:30:47.880086 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.880065 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d43bff14-4240-430d-972d-605f24dcf10b-config-file\") pod \"limitador-limitador-54d6d988d5-ksj9v\" (UID: \"d43bff14-4240-430d-972d-605f24dcf10b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:30:47.898054 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:47.898022 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh698\" (UniqueName: \"kubernetes.io/projected/d43bff14-4240-430d-972d-605f24dcf10b-kube-api-access-lh698\") pod \"limitador-limitador-54d6d988d5-ksj9v\" (UID: \"d43bff14-4240-430d-972d-605f24dcf10b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:30:48.001149 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:48.001124 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:30:48.191086 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:48.191055 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:30:48.193784 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:30:48.193758 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43bff14_4240_430d_972d_605f24dcf10b.slice/crio-3b8b0290e2bee59819fa24ff3d3bff5813c768dff74ef5b82971d51781727e1e WatchSource:0}: Error finding container 3b8b0290e2bee59819fa24ff3d3bff5813c768dff74ef5b82971d51781727e1e: Status 404 returned error can't find the container with id 3b8b0290e2bee59819fa24ff3d3bff5813c768dff74ef5b82971d51781727e1e Jul 06 12:30:48.556180 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:48.556147 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:48.813582 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:48.813505 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" event={"ID":"d43bff14-4240-430d-972d-605f24dcf10b","Type":"ContainerStarted","Data":"3b8b0290e2bee59819fa24ff3d3bff5813c768dff74ef5b82971d51781727e1e"} Jul 06 12:30:48.862069 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:48.862033 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:30:49.058211 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:49.058179 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-8xffw"] Jul 06 12:30:49.061550 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:49.061529 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8xffw" Jul 06 12:30:49.065533 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:49.065481 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vgnkc\"" Jul 06 12:30:49.078355 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:49.078332 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8xffw"] Jul 06 12:30:49.192042 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:49.192002 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmhqf\" (UniqueName: \"kubernetes.io/projected/de83fbeb-758a-48e0-9b76-a3783cef0768-kube-api-access-nmhqf\") pod \"authorino-75d556bd77-8xffw\" (UID: \"de83fbeb-758a-48e0-9b76-a3783cef0768\") " pod="kuadrant-system/authorino-75d556bd77-8xffw" Jul 06 12:30:49.293323 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:49.293291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhqf\" (UniqueName: \"kubernetes.io/projected/de83fbeb-758a-48e0-9b76-a3783cef0768-kube-api-access-nmhqf\") pod \"authorino-75d556bd77-8xffw\" (UID: \"de83fbeb-758a-48e0-9b76-a3783cef0768\") " pod="kuadrant-system/authorino-75d556bd77-8xffw" Jul 06 12:30:49.307125 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:49.307094 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmhqf\" (UniqueName: \"kubernetes.io/projected/de83fbeb-758a-48e0-9b76-a3783cef0768-kube-api-access-nmhqf\") pod \"authorino-75d556bd77-8xffw\" (UID: \"de83fbeb-758a-48e0-9b76-a3783cef0768\") " pod="kuadrant-system/authorino-75d556bd77-8xffw" Jul 06 12:30:49.372905 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:49.372825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8xffw" Jul 06 12:30:50.724625 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:50.724605 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8xffw"] Jul 06 12:30:50.727343 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:30:50.727314 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde83fbeb_758a_48e0_9b76_a3783cef0768.slice/crio-060ae683bd410ef2cacf83d3ab0b2d7e3019a97a4e93d534c78d01f72aef111f WatchSource:0}: Error finding container 060ae683bd410ef2cacf83d3ab0b2d7e3019a97a4e93d534c78d01f72aef111f: Status 404 returned error can't find the container with id 060ae683bd410ef2cacf83d3ab0b2d7e3019a97a4e93d534c78d01f72aef111f Jul 06 12:30:50.824898 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:50.824867 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" event={"ID":"d43bff14-4240-430d-972d-605f24dcf10b","Type":"ContainerStarted","Data":"20cf81cc42ad3e8d349650f673be6daa412ec5f25e3dd54808c0a142cfea5dd2"} Jul 06 12:30:50.825056 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:50.824938 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:30:50.825982 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:50.825958 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8xffw" event={"ID":"de83fbeb-758a-48e0-9b76-a3783cef0768","Type":"ContainerStarted","Data":"060ae683bd410ef2cacf83d3ab0b2d7e3019a97a4e93d534c78d01f72aef111f"} Jul 06 12:30:53.842972 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:53.842880 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8xffw" event={"ID":"de83fbeb-758a-48e0-9b76-a3783cef0768","Type":"ContainerStarted","Data":"d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7"} Jul 06 12:30:53.875722 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:53.875671 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-8xffw" podStartSLOduration=2.068567149 podStartE2EDuration="4.875643419s" totalImagesPullingTime="2.80707627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:30:53.874833668 +0000 UTC m=+469.953381695" watchObservedRunningTime="2026-07-06 12:30:53.875643419 +0000 UTC m=+469.954191445" Jul 06 12:30:53.875970 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:30:53.875951 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" podStartSLOduration=4.420652296 podStartE2EDuration="6.875945736s" totalImagesPullingTime="2.45529344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:30:50.851372049 +0000 UTC m=+466.929920076" watchObservedRunningTime="2026-07-06 12:30:53.875945736 +0000 UTC m=+469.954493809" Jul 06 12:31:01.838300 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:01.838268 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ksj9v" Jul 06 12:31:03.232609 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:03.232579 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:03.329017 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:03.328989 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:03.773272 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:03.773239 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:12.044483 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:12.044446 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:12.135568 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:12.135536 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:20.003364 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:20.003331 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:22.011872 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.011830 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:22.111922 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.111889 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:22.204589 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.204558 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jlcpw"] Jul 06 12:31:22.212181 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.212164 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" Jul 06 12:31:22.214921 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.214893 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jlcpw"] Jul 06 12:31:22.256268 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.256233 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkx4w\" (UniqueName: \"kubernetes.io/projected/0660e6d5-6b35-4757-b314-6382a0db8b8d-kube-api-access-fkx4w\") pod \"authorino-5d4b67c446-jlcpw\" (UID: \"0660e6d5-6b35-4757-b314-6382a0db8b8d\") " pod="kuadrant-system/authorino-5d4b67c446-jlcpw" Jul 06 12:31:22.357553 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.357480 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkx4w\" (UniqueName: \"kubernetes.io/projected/0660e6d5-6b35-4757-b314-6382a0db8b8d-kube-api-access-fkx4w\") pod \"authorino-5d4b67c446-jlcpw\" (UID: \"0660e6d5-6b35-4757-b314-6382a0db8b8d\") " pod="kuadrant-system/authorino-5d4b67c446-jlcpw" Jul 06 12:31:22.367481 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.367452 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkx4w\" (UniqueName: \"kubernetes.io/projected/0660e6d5-6b35-4757-b314-6382a0db8b8d-kube-api-access-fkx4w\") pod \"authorino-5d4b67c446-jlcpw\" (UID: \"0660e6d5-6b35-4757-b314-6382a0db8b8d\") " pod="kuadrant-system/authorino-5d4b67c446-jlcpw" Jul 06 12:31:22.448397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.448363 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jlcpw"] Jul 06 12:31:22.448605 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.448592 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" Jul 06 12:31:22.475217 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.475186 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68c58c4549-k6vt4"] Jul 06 12:31:22.478557 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.478538 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c58c4549-k6vt4" Jul 06 12:31:22.486810 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.486776 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68c58c4549-k6vt4"] Jul 06 12:31:22.559217 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.559181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhknf\" (UniqueName: \"kubernetes.io/projected/6ee53473-d43d-46c3-8f09-6466548f2a3e-kube-api-access-bhknf\") pod \"authorino-68c58c4549-k6vt4\" (UID: \"6ee53473-d43d-46c3-8f09-6466548f2a3e\") " pod="kuadrant-system/authorino-68c58c4549-k6vt4" Jul 06 12:31:22.561651 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.561621 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68c58c4549-k6vt4"] Jul 06 12:31:22.561920 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:31:22.561896 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bhknf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-68c58c4549-k6vt4" podUID="6ee53473-d43d-46c3-8f09-6466548f2a3e" Jul 06 12:31:22.575386 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.575366 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jlcpw"] Jul 06 12:31:22.577869 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:31:22.577844 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0660e6d5_6b35_4757_b314_6382a0db8b8d.slice/crio-bad457349a17a580fbc3519cfaaf5a331dc5b8a62c736e7934d4cd6cef6ff823 WatchSource:0}: Error finding container bad457349a17a580fbc3519cfaaf5a331dc5b8a62c736e7934d4cd6cef6ff823: Status 404 returned error can't find the container with id bad457349a17a580fbc3519cfaaf5a331dc5b8a62c736e7934d4cd6cef6ff823 Jul 06 12:31:22.592395 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.592371 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b5fd69d97-kxrl8"] Jul 06 12:31:22.595789 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.595773 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:31:22.598917 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.598899 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 12:31:22.608044 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.607997 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b5fd69d97-kxrl8"] Jul 06 12:31:22.660009 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.659978 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rk9h\" (UniqueName: \"kubernetes.io/projected/366f50bd-9565-4cd6-bce7-61598007fff1-kube-api-access-2rk9h\") pod \"authorino-6b5fd69d97-kxrl8\" (UID: \"366f50bd-9565-4cd6-bce7-61598007fff1\") " pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:31:22.660154 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.660042 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/366f50bd-9565-4cd6-bce7-61598007fff1-tls-cert\") pod \"authorino-6b5fd69d97-kxrl8\" (UID: \"366f50bd-9565-4cd6-bce7-61598007fff1\") " pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:31:22.660154 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.660082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhknf\" (UniqueName: \"kubernetes.io/projected/6ee53473-d43d-46c3-8f09-6466548f2a3e-kube-api-access-bhknf\") pod \"authorino-68c58c4549-k6vt4\" (UID: \"6ee53473-d43d-46c3-8f09-6466548f2a3e\") " pod="kuadrant-system/authorino-68c58c4549-k6vt4" Jul 06 12:31:22.669313 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.669292 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhknf\" (UniqueName: \"kubernetes.io/projected/6ee53473-d43d-46c3-8f09-6466548f2a3e-kube-api-access-bhknf\") pod \"authorino-68c58c4549-k6vt4\" (UID: \"6ee53473-d43d-46c3-8f09-6466548f2a3e\") " pod="kuadrant-system/authorino-68c58c4549-k6vt4" Jul 06 12:31:22.760724 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.760690 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/366f50bd-9565-4cd6-bce7-61598007fff1-tls-cert\") pod \"authorino-6b5fd69d97-kxrl8\" (UID: \"366f50bd-9565-4cd6-bce7-61598007fff1\") " pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:31:22.760878 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.760745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rk9h\" (UniqueName: \"kubernetes.io/projected/366f50bd-9565-4cd6-bce7-61598007fff1-kube-api-access-2rk9h\") pod \"authorino-6b5fd69d97-kxrl8\" (UID: \"366f50bd-9565-4cd6-bce7-61598007fff1\") " pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:31:22.763045 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.763023 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/366f50bd-9565-4cd6-bce7-61598007fff1-tls-cert\") pod \"authorino-6b5fd69d97-kxrl8\" (UID: \"366f50bd-9565-4cd6-bce7-61598007fff1\") " pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:31:22.771777 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.771750 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rk9h\" (UniqueName: \"kubernetes.io/projected/366f50bd-9565-4cd6-bce7-61598007fff1-kube-api-access-2rk9h\") pod \"authorino-6b5fd69d97-kxrl8\" (UID: \"366f50bd-9565-4cd6-bce7-61598007fff1\") " pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:31:22.904835 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.904775 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:31:22.965904 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.965873 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" event={"ID":"0660e6d5-6b35-4757-b314-6382a0db8b8d","Type":"ContainerStarted","Data":"bad457349a17a580fbc3519cfaaf5a331dc5b8a62c736e7934d4cd6cef6ff823"} Jul 06 12:31:22.965998 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.965906 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c58c4549-k6vt4" Jul 06 12:31:22.975068 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:22.975042 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c58c4549-k6vt4" Jul 06 12:31:23.052053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.052014 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b5fd69d97-kxrl8"] Jul 06 12:31:23.063247 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.063225 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhknf\" (UniqueName: \"kubernetes.io/projected/6ee53473-d43d-46c3-8f09-6466548f2a3e-kube-api-access-bhknf\") pod \"6ee53473-d43d-46c3-8f09-6466548f2a3e\" (UID: \"6ee53473-d43d-46c3-8f09-6466548f2a3e\") " Jul 06 12:31:23.065315 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.065282 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee53473-d43d-46c3-8f09-6466548f2a3e-kube-api-access-bhknf" (OuterVolumeSpecName: "kube-api-access-bhknf") pod "6ee53473-d43d-46c3-8f09-6466548f2a3e" (UID: "6ee53473-d43d-46c3-8f09-6466548f2a3e"). InnerVolumeSpecName "kube-api-access-bhknf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:31:23.164487 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.164417 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhknf\" (UniqueName: \"kubernetes.io/projected/6ee53473-d43d-46c3-8f09-6466548f2a3e-kube-api-access-bhknf\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:31:23.971174 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.971144 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" event={"ID":"366f50bd-9565-4cd6-bce7-61598007fff1","Type":"ContainerStarted","Data":"7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66"} Jul 06 12:31:23.971344 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.971189 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" event={"ID":"366f50bd-9565-4cd6-bce7-61598007fff1","Type":"ContainerStarted","Data":"0da13ca4c4cd50a542a4e23a02a9868ce8baa8521021f7debd3d635bcea4a30c"} Jul 06 12:31:23.972571 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.972535 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" event={"ID":"0660e6d5-6b35-4757-b314-6382a0db8b8d","Type":"ContainerStarted","Data":"667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f"} Jul 06 12:31:23.972571 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.972546 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c58c4549-k6vt4" Jul 06 12:31:23.972735 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.972610 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" podUID="0660e6d5-6b35-4757-b314-6382a0db8b8d" containerName="authorino" containerID="cri-o://667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f" gracePeriod=30 Jul 06 12:31:23.995653 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:23.995610 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" podStartSLOduration=1.65927551 podStartE2EDuration="1.995587429s" totalImagesPullingTime="336.311919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:31:23.993840123 +0000 UTC m=+500.072388151" watchObservedRunningTime="2026-07-06 12:31:23.995587429 +0000 UTC m=+500.074135434" Jul 06 12:31:24.016422 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.016378 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" podStartSLOduration=1.680632549 podStartE2EDuration="2.016364921s" totalImagesPullingTime="335.732372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:31:24.015398564 +0000 UTC m=+500.093946592" watchObservedRunningTime="2026-07-06 12:31:24.016364921 +0000 UTC m=+500.094912947" Jul 06 12:31:24.033180 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.033151 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8xffw"] Jul 06 12:31:24.033371 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.033347 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-8xffw" podUID="de83fbeb-758a-48e0-9b76-a3783cef0768" containerName="authorino" containerID="cri-o://d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7" gracePeriod=30 Jul 06 12:31:24.064640 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.064610 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68c58c4549-k6vt4"] Jul 06 12:31:24.067127 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.067101 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68c58c4549-k6vt4"] Jul 06 12:31:24.280374 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.280333 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" Jul 06 12:31:24.287210 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.287187 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8xffw" Jul 06 12:31:24.373229 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.373199 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkx4w\" (UniqueName: \"kubernetes.io/projected/0660e6d5-6b35-4757-b314-6382a0db8b8d-kube-api-access-fkx4w\") pod \"0660e6d5-6b35-4757-b314-6382a0db8b8d\" (UID: \"0660e6d5-6b35-4757-b314-6382a0db8b8d\") " Jul 06 12:31:24.373371 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.373311 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmhqf\" (UniqueName: \"kubernetes.io/projected/de83fbeb-758a-48e0-9b76-a3783cef0768-kube-api-access-nmhqf\") pod \"de83fbeb-758a-48e0-9b76-a3783cef0768\" (UID: \"de83fbeb-758a-48e0-9b76-a3783cef0768\") " Jul 06 12:31:24.375579 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.375551 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0660e6d5-6b35-4757-b314-6382a0db8b8d-kube-api-access-fkx4w" (OuterVolumeSpecName: "kube-api-access-fkx4w") pod "0660e6d5-6b35-4757-b314-6382a0db8b8d" (UID: "0660e6d5-6b35-4757-b314-6382a0db8b8d"). InnerVolumeSpecName "kube-api-access-fkx4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:31:24.375702 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.375614 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de83fbeb-758a-48e0-9b76-a3783cef0768-kube-api-access-nmhqf" (OuterVolumeSpecName: "kube-api-access-nmhqf") pod "de83fbeb-758a-48e0-9b76-a3783cef0768" (UID: "de83fbeb-758a-48e0-9b76-a3783cef0768"). InnerVolumeSpecName "kube-api-access-nmhqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:31:24.474372 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.474348 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkx4w\" (UniqueName: \"kubernetes.io/projected/0660e6d5-6b35-4757-b314-6382a0db8b8d-kube-api-access-fkx4w\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:31:24.474372 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.474372 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmhqf\" (UniqueName: \"kubernetes.io/projected/de83fbeb-758a-48e0-9b76-a3783cef0768-kube-api-access-nmhqf\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:31:24.977998 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.977959 2572 generic.go:356] "Generic (PLEG): container finished" podID="de83fbeb-758a-48e0-9b76-a3783cef0768" containerID="d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7" exitCode=0 Jul 06 12:31:24.978162 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.978020 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8xffw" Jul 06 12:31:24.978162 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.978037 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8xffw" event={"ID":"de83fbeb-758a-48e0-9b76-a3783cef0768","Type":"ContainerDied","Data":"d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7"} Jul 06 12:31:24.978162 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.978076 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8xffw" event={"ID":"de83fbeb-758a-48e0-9b76-a3783cef0768","Type":"ContainerDied","Data":"060ae683bd410ef2cacf83d3ab0b2d7e3019a97a4e93d534c78d01f72aef111f"} Jul 06 12:31:24.978162 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.978092 2572 scope.go:117] "RemoveContainer" containerID="d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7" Jul 06 12:31:24.979434 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.979376 2572 generic.go:356] "Generic (PLEG): container finished" podID="0660e6d5-6b35-4757-b314-6382a0db8b8d" containerID="667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f" exitCode=0 Jul 06 12:31:24.979492 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.979431 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" Jul 06 12:31:24.979492 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.979443 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" event={"ID":"0660e6d5-6b35-4757-b314-6382a0db8b8d","Type":"ContainerDied","Data":"667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f"} Jul 06 12:31:24.979579 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.979479 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jlcpw" event={"ID":"0660e6d5-6b35-4757-b314-6382a0db8b8d","Type":"ContainerDied","Data":"bad457349a17a580fbc3519cfaaf5a331dc5b8a62c736e7934d4cd6cef6ff823"} Jul 06 12:31:24.986811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.986794 2572 scope.go:117] "RemoveContainer" containerID="d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7" Jul 06 12:31:24.987097 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:31:24.987060 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7\": container with ID starting with d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7 not found: ID does not exist" containerID="d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7" Jul 06 12:31:24.987180 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.987096 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7"} err="failed to get container status \"d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7\": rpc error: code = NotFound desc = could not find container \"d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7\": container with ID starting with d950936a704c86bafbbef3b4c44891e221d3bafd87d751745b3d15f43dbf21d7 not found: ID does not exist" Jul 06 12:31:24.987180 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.987118 2572 scope.go:117] "RemoveContainer" containerID="667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f" Jul 06 12:31:24.994795 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.994779 2572 scope.go:117] "RemoveContainer" containerID="667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f" Jul 06 12:31:24.995047 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:31:24.995029 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f\": container with ID starting with 667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f not found: ID does not exist" containerID="667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f" Jul 06 12:31:24.995110 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:24.995053 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f"} err="failed to get container status \"667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f\": rpc error: code = NotFound desc = could not find container \"667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f\": container with ID starting with 667653d8c2b4d2facbe983b6df6d2a5c48c8f912c4ef65af822f41a1ca61f54f not found: ID does not exist" Jul 06 12:31:25.004704 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:25.004681 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8xffw"] Jul 06 12:31:25.009088 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:25.009068 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8xffw"] Jul 06 12:31:25.021741 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:25.021718 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jlcpw"] Jul 06 12:31:25.025797 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:25.025776 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jlcpw"] Jul 06 12:31:25.831398 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:25.831367 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0660e6d5-6b35-4757-b314-6382a0db8b8d" path="/var/lib/kubelet/pods/0660e6d5-6b35-4757-b314-6382a0db8b8d/volumes" Jul 06 12:31:25.831791 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:25.831704 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ee53473-d43d-46c3-8f09-6466548f2a3e" path="/var/lib/kubelet/pods/6ee53473-d43d-46c3-8f09-6466548f2a3e/volumes" Jul 06 12:31:25.831897 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:25.831885 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de83fbeb-758a-48e0-9b76-a3783cef0768" path="/var/lib/kubelet/pods/de83fbeb-758a-48e0-9b76-a3783cef0768/volumes" Jul 06 12:31:28.561397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:28.561366 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:28.666079 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:28.666046 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:28.806174 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:28.806145 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:28.969053 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:28.969023 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:31.558943 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:31.558906 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:31:31.689281 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:31:31.689251 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:03.190142 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:03.190069 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:03.678062 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:03.678026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:03.935922 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:03.935828 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:04.705227 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:04.705188 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:04.854928 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:04.854895 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:05.111601 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:05.111573 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:32:14.392352 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:14.392291 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:14.477644 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:14.477606 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:25.034177 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:25.034136 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:25.570514 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:25.570478 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:32:33.822729 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:33.822683 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:34.947394 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:34.947349 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:32:46.694674 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:46.694617 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:47.628766 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:47.628729 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:32:54.413698 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:54.413624 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:56.024866 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:56.024832 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:32:58.369523 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.369489 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678"] Jul 06 12:32:58.369902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.369808 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0660e6d5-6b35-4757-b314-6382a0db8b8d" containerName="authorino" Jul 06 12:32:58.369902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.369820 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0660e6d5-6b35-4757-b314-6382a0db8b8d" containerName="authorino" Jul 06 12:32:58.369902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.369829 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de83fbeb-758a-48e0-9b76-a3783cef0768" containerName="authorino" Jul 06 12:32:58.369902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.369834 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de83fbeb-758a-48e0-9b76-a3783cef0768" containerName="authorino" Jul 06 12:32:58.369902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.369886 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0660e6d5-6b35-4757-b314-6382a0db8b8d" containerName="authorino" Jul 06 12:32:58.369902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.369894 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="de83fbeb-758a-48e0-9b76-a3783cef0768" containerName="authorino" Jul 06 12:32:58.372549 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.372533 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.380929 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.380907 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 12:32:58.381065 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.380908 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 12:32:58.382418 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.382402 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lkt9g\"" Jul 06 12:32:58.382932 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.382919 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 12:32:58.401850 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.401827 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678"] Jul 06 12:32:58.422720 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.422654 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:32:58.450567 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.450540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.450700 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.450581 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.450700 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.450609 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.450700 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.450649 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.450700 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.450697 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qvmn\" (UniqueName: \"kubernetes.io/projected/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-kube-api-access-2qvmn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.450868 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.450746 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.551487 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.551441 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.551717 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.551502 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.551717 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.551528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.551717 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.551562 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.551717 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.551594 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvmn\" (UniqueName: \"kubernetes.io/projected/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-kube-api-access-2qvmn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.551717 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.551642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.552001 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.551943 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.552068 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.552000 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.552122 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.552071 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.554011 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.553984 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.554125 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.554111 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.579876 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.579849 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvmn\" (UniqueName: \"kubernetes.io/projected/58d8239f-b074-4c14-b9d1-ab3ac882a7c7-kube-api-access-2qvmn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nv678\" (UID: \"58d8239f-b074-4c14-b9d1-ab3ac882a7c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.683259 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.683177 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:32:58.826497 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:58.826343 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678"] Jul 06 12:32:58.829142 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:32:58.829107 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d8239f_b074_4c14_b9d1_ab3ac882a7c7.slice/crio-688e00d3d30c433be1337dcd1387f182c157ce1022d6bba534fb7a3df87de809 WatchSource:0}: Error finding container 688e00d3d30c433be1337dcd1387f182c157ce1022d6bba534fb7a3df87de809: Status 404 returned error can't find the container with id 688e00d3d30c433be1337dcd1387f182c157ce1022d6bba534fb7a3df87de809 Jul 06 12:32:59.385257 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:32:59.385174 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" event={"ID":"58d8239f-b074-4c14-b9d1-ab3ac882a7c7","Type":"ContainerStarted","Data":"688e00d3d30c433be1337dcd1387f182c157ce1022d6bba534fb7a3df87de809"} Jul 06 12:33:00.391769 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:00.391728 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" event={"ID":"58d8239f-b074-4c14-b9d1-ab3ac882a7c7","Type":"ContainerStarted","Data":"c92f0e432cbf4191ff2c6e93dcc0dddc56cb328d836295363fb0a58fa7cfcef2"} Jul 06 12:33:01.547289 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:01.547238 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" podStartSLOduration=2.525316652 podStartE2EDuration="3.547219214s" totalImagesPullingTime="1.021902562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:33:00.453484765 +0000 UTC m=+596.532032792" watchObservedRunningTime="2026-07-06 12:33:01.547219214 +0000 UTC m=+597.625767244" Jul 06 12:33:01.547710 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:01.547432 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:33:08.072174 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:08.072139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:33:08.684217 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:08.684184 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:33:08.684387 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:08.684245 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:33:08.696715 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:08.696681 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:33:09.442844 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:09.442815 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nv678" Jul 06 12:33:10.414127 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:10.414093 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:33:44.292630 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:44.292593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:33:49.866175 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:33:49.866141 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:34:09.629521 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.629482 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-774458dd6d-r66pd"] Jul 06 12:34:09.631836 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.631820 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774458dd6d-r66pd" Jul 06 12:34:09.644128 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.644103 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-774458dd6d-r66pd"] Jul 06 12:34:09.754840 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.754797 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe4b7728-a734-44d7-93bd-6a97b8a70cc1-tls-cert\") pod \"authorino-774458dd6d-r66pd\" (UID: \"fe4b7728-a734-44d7-93bd-6a97b8a70cc1\") " pod="kuadrant-system/authorino-774458dd6d-r66pd" Jul 06 12:34:09.755007 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.754864 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78skm\" (UniqueName: \"kubernetes.io/projected/fe4b7728-a734-44d7-93bd-6a97b8a70cc1-kube-api-access-78skm\") pod \"authorino-774458dd6d-r66pd\" (UID: \"fe4b7728-a734-44d7-93bd-6a97b8a70cc1\") " pod="kuadrant-system/authorino-774458dd6d-r66pd" Jul 06 12:34:09.855963 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.855934 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe4b7728-a734-44d7-93bd-6a97b8a70cc1-tls-cert\") pod \"authorino-774458dd6d-r66pd\" (UID: \"fe4b7728-a734-44d7-93bd-6a97b8a70cc1\") " pod="kuadrant-system/authorino-774458dd6d-r66pd" Jul 06 12:34:09.856152 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.855988 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78skm\" (UniqueName: \"kubernetes.io/projected/fe4b7728-a734-44d7-93bd-6a97b8a70cc1-kube-api-access-78skm\") pod \"authorino-774458dd6d-r66pd\" (UID: \"fe4b7728-a734-44d7-93bd-6a97b8a70cc1\") " pod="kuadrant-system/authorino-774458dd6d-r66pd" Jul 06 12:34:09.858371 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.858348 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe4b7728-a734-44d7-93bd-6a97b8a70cc1-tls-cert\") pod \"authorino-774458dd6d-r66pd\" (UID: \"fe4b7728-a734-44d7-93bd-6a97b8a70cc1\") " pod="kuadrant-system/authorino-774458dd6d-r66pd" Jul 06 12:34:09.867491 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.867470 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78skm\" (UniqueName: \"kubernetes.io/projected/fe4b7728-a734-44d7-93bd-6a97b8a70cc1-kube-api-access-78skm\") pod \"authorino-774458dd6d-r66pd\" (UID: \"fe4b7728-a734-44d7-93bd-6a97b8a70cc1\") " pod="kuadrant-system/authorino-774458dd6d-r66pd" Jul 06 12:34:09.941027 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.940961 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774458dd6d-r66pd" Jul 06 12:34:09.975381 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:09.975351 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:34:10.079253 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:10.079226 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-774458dd6d-r66pd"] Jul 06 12:34:10.080139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:10.080114 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:34:10.081118 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:34:10.081090 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe4b7728_a734_44d7_93bd_6a97b8a70cc1.slice/crio-6d0c92704c61adcb35cf4ac2a00f8368dbbd42a7585b0c0c56bcbbfb76c601b3 WatchSource:0}: Error finding container 6d0c92704c61adcb35cf4ac2a00f8368dbbd42a7585b0c0c56bcbbfb76c601b3: Status 404 returned error can't find the container with id 6d0c92704c61adcb35cf4ac2a00f8368dbbd42a7585b0c0c56bcbbfb76c601b3 Jul 06 12:34:10.082646 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:10.082630 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 12:34:10.681813 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:10.681779 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774458dd6d-r66pd" event={"ID":"fe4b7728-a734-44d7-93bd-6a97b8a70cc1","Type":"ContainerStarted","Data":"342d39fa242485d7ce5344788e096acfc50e757596cdef9b94e0cf5936d42fc9"} Jul 06 12:34:10.681813 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:10.681818 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774458dd6d-r66pd" event={"ID":"fe4b7728-a734-44d7-93bd-6a97b8a70cc1","Type":"ContainerStarted","Data":"6d0c92704c61adcb35cf4ac2a00f8368dbbd42a7585b0c0c56bcbbfb76c601b3"} Jul 06 12:34:10.726377 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:10.726325 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-774458dd6d-r66pd" podStartSLOduration=1.354486165 podStartE2EDuration="1.726306826s" totalImagesPullingTime="371.820661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:34:10.725202543 +0000 UTC m=+666.803750572" watchObservedRunningTime="2026-07-06 12:34:10.726306826 +0000 UTC m=+666.804854864" Jul 06 12:34:10.793056 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:10.793019 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b5fd69d97-kxrl8"] Jul 06 12:34:10.793222 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:10.793199 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" podUID="366f50bd-9565-4cd6-bce7-61598007fff1" containerName="authorino" containerID="cri-o://7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66" gracePeriod=30 Jul 06 12:34:11.051145 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.051115 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:34:11.166601 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.166561 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rk9h\" (UniqueName: \"kubernetes.io/projected/366f50bd-9565-4cd6-bce7-61598007fff1-kube-api-access-2rk9h\") pod \"366f50bd-9565-4cd6-bce7-61598007fff1\" (UID: \"366f50bd-9565-4cd6-bce7-61598007fff1\") " Jul 06 12:34:11.166801 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.166781 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/366f50bd-9565-4cd6-bce7-61598007fff1-tls-cert\") pod \"366f50bd-9565-4cd6-bce7-61598007fff1\" (UID: \"366f50bd-9565-4cd6-bce7-61598007fff1\") " Jul 06 12:34:11.178710 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.178648 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366f50bd-9565-4cd6-bce7-61598007fff1-kube-api-access-2rk9h" (OuterVolumeSpecName: "kube-api-access-2rk9h") pod "366f50bd-9565-4cd6-bce7-61598007fff1" (UID: "366f50bd-9565-4cd6-bce7-61598007fff1"). InnerVolumeSpecName "kube-api-access-2rk9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:34:11.181640 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.181608 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366f50bd-9565-4cd6-bce7-61598007fff1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "366f50bd-9565-4cd6-bce7-61598007fff1" (UID: "366f50bd-9565-4cd6-bce7-61598007fff1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:34:11.212383 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.212320 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:34:11.267569 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.267540 2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/366f50bd-9565-4cd6-bce7-61598007fff1-tls-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:34:11.267569 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.267568 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rk9h\" (UniqueName: \"kubernetes.io/projected/366f50bd-9565-4cd6-bce7-61598007fff1-kube-api-access-2rk9h\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:34:11.332379 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.332260 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:34:11.688519 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.688436 2572 generic.go:356] "Generic (PLEG): container finished" podID="366f50bd-9565-4cd6-bce7-61598007fff1" containerID="7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66" exitCode=0 Jul 06 12:34:11.688519 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.688482 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" Jul 06 12:34:11.689021 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.688517 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" event={"ID":"366f50bd-9565-4cd6-bce7-61598007fff1","Type":"ContainerDied","Data":"7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66"} Jul 06 12:34:11.689021 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.688561 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b5fd69d97-kxrl8" event={"ID":"366f50bd-9565-4cd6-bce7-61598007fff1","Type":"ContainerDied","Data":"0da13ca4c4cd50a542a4e23a02a9868ce8baa8521021f7debd3d635bcea4a30c"} Jul 06 12:34:11.689021 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.688583 2572 scope.go:117] "RemoveContainer" containerID="7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66" Jul 06 12:34:11.697647 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.697630 2572 scope.go:117] "RemoveContainer" containerID="7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66" Jul 06 12:34:11.697914 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:34:11.697898 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66\": container with ID starting with 7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66 not found: ID does not exist" containerID="7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66" Jul 06 12:34:11.698020 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.697926 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66"} err="failed to get container status \"7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66\": rpc error: code = NotFound desc = could not find container \"7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66\": container with ID starting with 7c3fbc369c02009c21fdae7ae26569fd5f597cdb7d480ef6a1962bb64c188b66 not found: ID does not exist" Jul 06 12:34:11.713781 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.713757 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b5fd69d97-kxrl8"] Jul 06 12:34:11.718105 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.718070 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b5fd69d97-kxrl8"] Jul 06 12:34:11.833421 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:11.833382 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="366f50bd-9565-4cd6-bce7-61598007fff1" path="/var/lib/kubelet/pods/366f50bd-9565-4cd6-bce7-61598007fff1/volumes" Jul 06 12:34:17.202287 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:17.202248 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:34:17.343009 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:17.342977 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:34:40.236006 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:40.235960 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:34:48.610589 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:48.610552 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:34:51.238645 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:34:51.238608 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:35:02.778890 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:35:02.778810 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:35:10.823111 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:35:10.823075 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:35:22.209486 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:35:22.209450 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:35:24.302433 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:35:24.302395 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:35:25.734713 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:35:25.734679 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:35:29.955378 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:35:29.955344 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:35:48.633274 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:35:48.633243 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:36:00.823143 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:00.823107 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:36:06.460224 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:06.460189 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:36:06.544215 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:06.544180 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:36:06.653014 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:06.652981 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:36:13.345219 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:13.345184 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:36:13.477360 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:13.477324 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:36:13.577163 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:13.577130 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:36:13.734504 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:13.734473 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:36:20.711169 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:20.711130 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:36:20.809776 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:20.809741 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:36:38.507036 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:38.506996 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:36:47.253505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:47.253468 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:36:57.615953 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:36:57.615915 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:37:09.316605 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:37:09.316573 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:37:18.641946 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:37:18.641909 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:37:34.734279 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:37:34.734244 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:37:40.825247 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:37:40.825208 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:37:49.418809 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:37:49.418772 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:38:12.024936 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:38:12.024900 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:38:24.823184 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:38:24.823149 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:38:49.592521 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:38:49.592482 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:39:04.719723 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:39:04.719679 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:39:07.612564 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:39:07.612524 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:39:16.556708 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:39:16.556672 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:39:26.834577 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:39:26.834499 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:39:28.259721 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:39:28.259620 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:39:36.215379 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:39:36.215347 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:39:45.216015 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:39:45.215979 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:39:53.811328 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:39:53.811293 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:41:15.386796 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:41:15.386715 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:41:43.117416 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:41:43.117381 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:41:52.105444 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:41:52.105405 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:42:05.036426 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:42:05.036389 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:42:16.662114 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:42:16.662078 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:42:29.304349 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:42:29.304254 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:42:38.223156 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:42:38.223121 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:42:47.333418 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:42:47.333381 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:42:57.821745 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:42:57.821713 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:43:09.418915 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:43:09.418877 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:43:18.210784 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:43:18.210745 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:43:20.383463 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:43:20.383430 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:43:31.028154 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:43:31.028116 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:43:55.722532 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:43:55.722495 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:44:03.016375 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:44:03.016340 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:44:12.337242 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:44:12.337211 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:44:27.812440 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:44:27.812405 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:44:43.215077 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:44:43.215041 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:44:53.406634 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:44:53.406599 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:45:02.120642 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:45:02.120608 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:45:12.827803 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:45:12.827767 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:45:22.119381 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:45:22.119345 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:45:30.813938 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:45:30.813856 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:45:45.314049 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:45:45.314005 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:45:55.026415 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:45:55.026371 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:46:04.110981 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:46:04.110941 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:46:15.312269 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:46:15.312231 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:46:28.108218 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:46:28.108183 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:46:36.643841 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:46:36.643803 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:46:49.392618 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:46:49.392585 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:47:01.533435 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:01.533343 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:47:04.509536 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:04.509505 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:12.026785 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:12.026751 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:47:14.493253 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:14.493215 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:22.040195 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:22.040158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:24.906538 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:24.906498 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:47:30.452538 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:30.452502 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:35.426288 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:35.426233 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:35.774479 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:35.774440 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:38.414863 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:38.414806 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:38.549571 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:38.549526 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:38.947941 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:38.947902 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:39.265307 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:39.265275 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:47:45.006876 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:45.006837 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:45.109239 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:45.109209 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:45.304822 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:45.304739 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:45.404210 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:45.404174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:46.453958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:46.453917 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:46.547351 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:46.547299 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:46.680130 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:46.680095 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:47.146842 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:47.146809 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:47.758904 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:47.758859 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:47.877333 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:47.877297 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:48.696392 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:48.696337 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:49.742011 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:49.741980 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:47:51.665957 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:51.665918 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:51.833612 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:51.833567 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:51.929165 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:51.929074 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:52.042016 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:52.041981 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:53.784464 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:53.784427 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:57.129006 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:57.128966 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:58.493491 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:58.493436 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:58.599285 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:58.599254 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:58.753948 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:58.753856 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:58.846793 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:47:58.846764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:00.120056 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:00.120009 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:00.249045 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:00.249004 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:00.347111 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:00.347080 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:00.532698 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:00.532648 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:00.898537 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:00.898443 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:01.000346 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:01.000314 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:01.843570 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:01.843530 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:02.402853 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:02.402821 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:48:05.935127 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:05.935097 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:06.035977 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:06.035943 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:11.321442 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:11.321406 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:11.421957 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:11.421921 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:15.509752 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:15.509713 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:18.124059 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:18.124029 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:18.213396 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:18.213259 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:18.394079 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:18.394003 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:18.489275 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:18.489242 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:19.589359 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:19.589316 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:19.696856 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:19.696820 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:19.798875 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:19.798840 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:19.905253 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:19.905172 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:20.277548 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:20.277501 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:20.379451 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:20.379416 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:21.261561 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:21.261526 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:24.081978 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:24.081946 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:24.223720 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:24.223684 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:24.367214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:24.367126 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:24.556522 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:24.556482 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:25.977408 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:25.977369 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:30.541937 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:30.541903 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:30.617850 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:30.617817 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:31.035361 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:31.035332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:31.145841 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:31.145789 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:32.671885 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:32.671850 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:37.073606 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:37.073547 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:37.174201 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:37.174166 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:39.227440 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:39.227405 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:39.349371 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:39.349332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:39.445087 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:39.445042 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:39.547645 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:39.547560 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:48:39.969627 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:48:39.969592 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:05.924404 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:05.924366 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:12.627130 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:12.627081 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:12.734682 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:12.734625 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:12.827836 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:12.827805 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:28.729786 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:28.729736 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:34.956244 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:34.956197 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:35.032972 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:35.032935 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:35.145214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:35.145181 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:35.241165 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:35.241130 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:45.765421 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:45.765377 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:46.564851 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:46.564817 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:52.912954 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:52.912906 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:53.072292 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:53.072255 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:53.203524 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:53.203439 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:53.330976 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:53.330940 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:54.031181 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:54.031149 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:54.167247 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:54.167207 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:49:55.326710 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:55.326677 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:49:55.871732 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:49:55.871693 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:02.163113 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:02.163069 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:02.233159 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:02.233117 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:02.511337 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:02.511302 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:50:02.596592 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:02.596556 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:02.901623 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:02.901529 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:09.115623 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:09.115579 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:09.216078 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:09.216018 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:09.405932 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:09.405838 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:09.680034 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:09.679950 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:50:09.826337 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:09.826303 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:09.924462 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:09.924424 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:12.841941 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:12.841904 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:12.988141 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:12.988093 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:13.145779 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:13.145695 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:13.545120 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:13.545088 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:13.754926 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:13.754888 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:13.861695 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:13.861575 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:17.304113 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:17.304070 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:17.983405 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:17.983370 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:18.084398 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:18.084360 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:18.447941 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:18.447848 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:19.314568 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:19.314529 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:50:24.786672 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:24.786626 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:24.846700 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:24.846634 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:25.241960 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:25.241924 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:25.597392 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:25.597313 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:25.695931 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:25.695894 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:29.331705 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:29.331649 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:50:31.622216 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:31.622176 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:31.721533 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:31.721493 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:31.888944 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:31.888853 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:32.233178 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:32.233145 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:32.330673 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:32.330638 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:32.458329 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:32.458296 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:38.568475 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:38.568437 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:38.672791 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:38.672764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:38.913024 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:38.912934 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:39.084883 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:39.084849 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:44.962022 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:44.961991 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:45.028209 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:45.028174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:48.539162 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:48.539124 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:48.638596 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:48.638560 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:51.793782 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:51.793743 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:52.135143 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:52.135063 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:52.232678 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:52.232630 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:54.926139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:54.926103 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:55.107559 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:55.107517 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:55.246417 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:55.246386 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:56.089212 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:56.089172 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:56.312333 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:56.312295 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:59.407137 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:59.407093 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:50:59.739323 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:50:59.739291 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:51:10.930073 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.930036 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns"] Jul 06 12:51:10.930570 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.930369 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366f50bd-9565-4cd6-bce7-61598007fff1" containerName="authorino" Jul 06 12:51:10.930570 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.930381 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="366f50bd-9565-4cd6-bce7-61598007fff1" containerName="authorino" Jul 06 12:51:10.930570 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.930431 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="366f50bd-9565-4cd6-bce7-61598007fff1" containerName="authorino" Jul 06 12:51:10.933174 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.933155 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:10.936309 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.936286 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0113cf1b\"/\"default-dockercfg-mhwdd\"" Jul 06 12:51:10.936420 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.936322 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0113cf1b\"/\"openshift-service-ca.crt\"" Jul 06 12:51:10.937798 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.937783 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0113cf1b\"/\"kube-root-ca.crt\"" Jul 06 12:51:10.937896 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.937856 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0113cf1b\"/\"e2e-lifecycle-model-0113cf1b-kserve-self-signed-certs\"" Jul 06 12:51:10.946546 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.946525 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns"] Jul 06 12:51:10.963902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:10.963870 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:51:11.064456 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.064416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-dshm\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.064648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.064479 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-home\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.064648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.064503 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh5w6\" (UniqueName: \"kubernetes.io/projected/83a3fdd4-5906-46f2-b26a-689b60313256-kube-api-access-qh5w6\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.064648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.064532 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-tmp-dir\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.064648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.064598 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a3fdd4-5906-46f2-b26a-689b60313256-tls-certs\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.064648 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.064621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-model-cache\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.165231 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.165189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-dshm\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.165422 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.165247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-home\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.165422 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.165275 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh5w6\" (UniqueName: \"kubernetes.io/projected/83a3fdd4-5906-46f2-b26a-689b60313256-kube-api-access-qh5w6\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.165422 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.165303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-tmp-dir\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.165422 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.165354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a3fdd4-5906-46f2-b26a-689b60313256-tls-certs\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.165422 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.165378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-model-cache\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.165749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.165712 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-home\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.165749 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.165732 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-tmp-dir\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.165848 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.165786 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-model-cache\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.167499 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.167480 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-dshm\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.167706 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.167689 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a3fdd4-5906-46f2-b26a-689b60313256-tls-certs\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.175811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.175791 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh5w6\" (UniqueName: \"kubernetes.io/projected/83a3fdd4-5906-46f2-b26a-689b60313256-kube-api-access-qh5w6\") pod \"e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.242840 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.242815 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:11.377810 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.377787 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns"] Jul 06 12:51:11.379547 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:51:11.379518 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83a3fdd4_5906_46f2_b26a_689b60313256.slice/crio-ce71b6e52ceaf978a53b90cf1df7460aabe7aab3528a4bbbe2e8d0f8d082b685 WatchSource:0}: Error finding container ce71b6e52ceaf978a53b90cf1df7460aabe7aab3528a4bbbe2e8d0f8d082b685: Status 404 returned error can't find the container with id ce71b6e52ceaf978a53b90cf1df7460aabe7aab3528a4bbbe2e8d0f8d082b685 Jul 06 12:51:11.908139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.908107 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" event={"ID":"83a3fdd4-5906-46f2-b26a-689b60313256","Type":"ContainerStarted","Data":"7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a"} Jul 06 12:51:11.908139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.908146 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" event={"ID":"83a3fdd4-5906-46f2-b26a-689b60313256","Type":"ContainerStarted","Data":"ce71b6e52ceaf978a53b90cf1df7460aabe7aab3528a4bbbe2e8d0f8d082b685"} Jul 06 12:51:11.928609 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:11.928560 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" podStartSLOduration=1.9285451390000001 podStartE2EDuration="1.928545139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:51:11.927754434 +0000 UTC m=+1688.006302474" watchObservedRunningTime="2026-07-06 12:51:11.928545139 +0000 UTC m=+1688.007093164" Jul 06 12:51:14.725501 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:14.725465 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:51:21.243633 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:21.243582 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:21.243633 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:21.243641 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:21.256207 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:21.256180 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:21.961466 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:21.961438 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:31.013127 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:31.013049 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:51:31.741809 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:31.741766 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:51:32.264883 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:32.264839 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:51:37.843709 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:37.843645 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:51:42.830223 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:42.830186 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns"] Jul 06 12:51:42.926621 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:42.926583 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:51:43.608316 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:43.608284 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:51:57.849653 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:57.849618 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" podUID="83a3fdd4-5906-46f2-b26a-689b60313256" containerName="main" containerID="cri-o://7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a" gracePeriod=45 Jul 06 12:51:58.089410 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.089379 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:58.102138 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.102076 2572 generic.go:356] "Generic (PLEG): container finished" podID="83a3fdd4-5906-46f2-b26a-689b60313256" containerID="7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a" exitCode=0 Jul 06 12:51:58.102253 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.102184 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" Jul 06 12:51:58.102298 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.102186 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" event={"ID":"83a3fdd4-5906-46f2-b26a-689b60313256","Type":"ContainerDied","Data":"7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a"} Jul 06 12:51:58.102356 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.102331 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns" event={"ID":"83a3fdd4-5906-46f2-b26a-689b60313256","Type":"ContainerDied","Data":"ce71b6e52ceaf978a53b90cf1df7460aabe7aab3528a4bbbe2e8d0f8d082b685"} Jul 06 12:51:58.102400 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.102376 2572 scope.go:117] "RemoveContainer" containerID="7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a" Jul 06 12:51:58.110703 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.110681 2572 scope.go:117] "RemoveContainer" containerID="7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a" Jul 06 12:51:58.111062 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:51:58.111030 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a\": container with ID starting with 7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a not found: ID does not exist" containerID="7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a" Jul 06 12:51:58.111149 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.111064 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a"} err="failed to get container status \"7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a\": rpc error: code = NotFound desc = could not find container \"7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a\": container with ID starting with 7041d4856039aa8bd4d41c43dac01f94a22f80b127946339305ffc178010de3a not found: ID does not exist" Jul 06 12:51:58.162538 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.162498 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-model-cache\") pod \"83a3fdd4-5906-46f2-b26a-689b60313256\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " Jul 06 12:51:58.162538 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.162535 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-home\") pod \"83a3fdd4-5906-46f2-b26a-689b60313256\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " Jul 06 12:51:58.162830 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.162570 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-dshm\") pod \"83a3fdd4-5906-46f2-b26a-689b60313256\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " Jul 06 12:51:58.162830 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.162599 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a3fdd4-5906-46f2-b26a-689b60313256-tls-certs\") pod \"83a3fdd4-5906-46f2-b26a-689b60313256\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " Jul 06 12:51:58.162830 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.162644 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-tmp-dir\") pod \"83a3fdd4-5906-46f2-b26a-689b60313256\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " Jul 06 12:51:58.162830 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.162716 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh5w6\" (UniqueName: \"kubernetes.io/projected/83a3fdd4-5906-46f2-b26a-689b60313256-kube-api-access-qh5w6\") pod \"83a3fdd4-5906-46f2-b26a-689b60313256\" (UID: \"83a3fdd4-5906-46f2-b26a-689b60313256\") " Jul 06 12:51:58.163043 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.162846 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-model-cache" (OuterVolumeSpecName: "model-cache") pod "83a3fdd4-5906-46f2-b26a-689b60313256" (UID: "83a3fdd4-5906-46f2-b26a-689b60313256"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:51:58.163043 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.162857 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-home" (OuterVolumeSpecName: "home") pod "83a3fdd4-5906-46f2-b26a-689b60313256" (UID: "83a3fdd4-5906-46f2-b26a-689b60313256"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:51:58.163043 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.162943 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83a3fdd4-5906-46f2-b26a-689b60313256" (UID: "83a3fdd4-5906-46f2-b26a-689b60313256"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:51:58.163043 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.163002 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:51:58.163043 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.163017 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:51:58.163043 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.163031 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:51:58.164822 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.164793 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-dshm" (OuterVolumeSpecName: "dshm") pod "83a3fdd4-5906-46f2-b26a-689b60313256" (UID: "83a3fdd4-5906-46f2-b26a-689b60313256"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:51:58.164922 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.164859 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a3fdd4-5906-46f2-b26a-689b60313256-kube-api-access-qh5w6" (OuterVolumeSpecName: "kube-api-access-qh5w6") pod "83a3fdd4-5906-46f2-b26a-689b60313256" (UID: "83a3fdd4-5906-46f2-b26a-689b60313256"). InnerVolumeSpecName "kube-api-access-qh5w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:51:58.164922 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.164858 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a3fdd4-5906-46f2-b26a-689b60313256-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83a3fdd4-5906-46f2-b26a-689b60313256" (UID: "83a3fdd4-5906-46f2-b26a-689b60313256"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:51:58.263721 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.263651 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh5w6\" (UniqueName: \"kubernetes.io/projected/83a3fdd4-5906-46f2-b26a-689b60313256-kube-api-access-qh5w6\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:51:58.263721 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.263719 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a3fdd4-5906-46f2-b26a-689b60313256-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:51:58.263721 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.263734 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a3fdd4-5906-46f2-b26a-689b60313256-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:51:58.425639 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.425612 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns"] Jul 06 12:51:58.429582 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:58.429557 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0113cf1b/e2e-lifecycle-model-0113cf1b-kserve-6f4f956587-wxgns"] Jul 06 12:51:59.831919 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:51:59.831885 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a3fdd4-5906-46f2-b26a-689b60313256" path="/var/lib/kubelet/pods/83a3fdd4-5906-46f2-b26a-689b60313256/volumes" Jul 06 12:52:19.537110 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:19.537077 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:21.110834 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:21.110802 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:22.506727 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:22.506697 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:52:27.711419 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:27.711382 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:42.655154 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:42.655119 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:42.755381 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:42.755348 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:42.974178 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:42.974141 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:44.268444 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:44.268405 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:44.365262 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:44.365230 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:44.469685 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:44.469639 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:45.316527 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:45.316488 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:45.404098 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:45.404062 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:46.723013 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:46.722975 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:52:46.859828 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:46.859772 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:53.081190 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:53.081156 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:53.180844 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:53.180809 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:53.755946 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:53.755912 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:52:59.987696 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:52:59.987589 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:00.109922 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:00.109887 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:00.197838 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:00.197797 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:00.808988 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:00.808950 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:53:10.707833 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:10.707797 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:11.036204 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:11.036101 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:31.986116 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:31.986079 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:38.102498 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:38.102467 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:38.272140 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:38.272103 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:38.435865 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:38.435780 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:38.793237 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:38.793198 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:39.039000 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:39.038967 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:41.653120 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:41.653077 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:41.750007 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:41.749977 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:42.025532 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:42.025495 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:42.268870 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:42.268831 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:42.635204 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:42.635173 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:42.736255 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:42.736208 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:46.112297 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:46.112258 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:46.433018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:46.432938 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:46.809534 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:46.809495 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:47.590558 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:47.590521 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:49.221855 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:49.221817 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:53:49.668935 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:49.668855 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:49.768172 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:49.768131 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:49.926084 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:49.925992 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:50.076083 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:50.076052 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:50.470204 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:50.470173 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:50.573208 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:50.573175 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:53:54.167888 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:53:54.167853 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:00.034012 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.033981 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2"] Jul 06 12:54:00.034394 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.034291 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a3fdd4-5906-46f2-b26a-689b60313256" containerName="main" Jul 06 12:54:00.034394 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.034304 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a3fdd4-5906-46f2-b26a-689b60313256" containerName="main" Jul 06 12:54:00.034394 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.034371 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a3fdd4-5906-46f2-b26a-689b60313256" containerName="main" Jul 06 12:54:00.037214 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.037198 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.040349 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.040328 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-86ab8b\"/\"default-dockercfg-4xscf\"" Jul 06 12:54:00.040537 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.040518 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-86ab8b\"/\"openshift-service-ca.crt\"" Jul 06 12:54:00.041805 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.041787 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-86ab8b\"/\"test-model-86ab8b-kserve-self-signed-certs\"" Jul 06 12:54:00.041901 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.041846 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-86ab8b\"/\"kube-root-ca.crt\"" Jul 06 12:54:00.052331 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.052310 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2"] Jul 06 12:54:00.078168 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.078143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-model-cache\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.078324 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.078183 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-dshm\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.078324 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.078213 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvhm\" (UniqueName: \"kubernetes.io/projected/4c0afae9-3c5d-4210-aa74-743906b1fd22-kube-api-access-vbvhm\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.078436 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.078327 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-tmp-dir\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.078436 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.078367 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c0afae9-3c5d-4210-aa74-743906b1fd22-tls-certs\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.078513 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.078450 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-home\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.162938 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.162892 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:00.179637 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.179602 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-dshm\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.179811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.179649 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvhm\" (UniqueName: \"kubernetes.io/projected/4c0afae9-3c5d-4210-aa74-743906b1fd22-kube-api-access-vbvhm\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.179811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.179708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-tmp-dir\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.179811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.179727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c0afae9-3c5d-4210-aa74-743906b1fd22-tls-certs\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.179811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.179762 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-home\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.179811 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.179796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-model-cache\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.180172 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.180141 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-tmp-dir\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.180294 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.180174 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-model-cache\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.180294 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.180215 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-home\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.181953 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.181929 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-dshm\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.182109 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.182090 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c0afae9-3c5d-4210-aa74-743906b1fd22-tls-certs\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.197194 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.197171 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvhm\" (UniqueName: \"kubernetes.io/projected/4c0afae9-3c5d-4210-aa74-743906b1fd22-kube-api-access-vbvhm\") pod \"test-model-86ab8b-kserve-b6fd4d555-488b2\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.346633 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.346551 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:00.485273 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.485232 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2"] Jul 06 12:54:00.487912 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:54:00.487882 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c0afae9_3c5d_4210_aa74_743906b1fd22.slice/crio-e205281603328789b27a3abb73ae11f2da3b4209fcbbebecb040f831ceeba241 WatchSource:0}: Error finding container e205281603328789b27a3abb73ae11f2da3b4209fcbbebecb040f831ceeba241: Status 404 returned error can't find the container with id e205281603328789b27a3abb73ae11f2da3b4209fcbbebecb040f831ceeba241 Jul 06 12:54:00.603602 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.603502 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" event={"ID":"4c0afae9-3c5d-4210-aa74-743906b1fd22","Type":"ContainerStarted","Data":"e2bae22f26377822ab26e0f214c7ad6b310bf00eec0b11f119df39d008b769b2"} Jul 06 12:54:00.603602 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.603545 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" event={"ID":"4c0afae9-3c5d-4210-aa74-743906b1fd22","Type":"ContainerStarted","Data":"e205281603328789b27a3abb73ae11f2da3b4209fcbbebecb040f831ceeba241"} Jul 06 12:54:00.624422 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:00.624376 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" podStartSLOduration=0.624360773 podStartE2EDuration="624.360773ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:54:00.622977508 +0000 UTC m=+1856.701525532" watchObservedRunningTime="2026-07-06 12:54:00.624360773 +0000 UTC m=+1856.702908799" Jul 06 12:54:03.492335 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:03.492297 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:03.592277 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:03.592217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:04.125171 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:04.125138 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:54:10.347370 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:10.347329 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:10.347820 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:10.347385 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:10.359875 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:10.359847 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:10.655541 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:10.655461 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:11.518962 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:11.518924 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:11.620009 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:11.619977 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:18.102703 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:18.102650 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:54:22.832575 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.832540 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz"] Jul 06 12:54:22.835810 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.835795 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:22.839653 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.839623 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c5c936\"/\"openshift-service-ca.crt\"" Jul 06 12:54:22.839653 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.839623 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c5c936\"/\"default-dockercfg-2v6j9\"" Jul 06 12:54:22.839875 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.839710 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c5c936\"/\"kube-root-ca.crt\"" Jul 06 12:54:22.839875 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.839623 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c5c936\"/\"test-model-c5c936-kserve-self-signed-certs\"" Jul 06 12:54:22.846183 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.846162 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz"] Jul 06 12:54:22.865085 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.865058 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:22.971851 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.971815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-tmp-dir\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:22.972018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.971873 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a85083-96ea-45e8-8574-fd047ddcb38b-tls-certs\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:22.972018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.971924 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-home\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:22.972018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.971942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-model-cache\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:22.972018 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.971969 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-dshm\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:22.972227 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:22.972065 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrdx\" (UniqueName: \"kubernetes.io/projected/25a85083-96ea-45e8-8574-fd047ddcb38b-kube-api-access-nfrdx\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.073535 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.073500 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-home\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.073535 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.073538 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-model-cache\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.073807 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.073556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-dshm\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.073807 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.073591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrdx\" (UniqueName: \"kubernetes.io/projected/25a85083-96ea-45e8-8574-fd047ddcb38b-kube-api-access-nfrdx\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.073807 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.073634 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-tmp-dir\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.073807 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.073710 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a85083-96ea-45e8-8574-fd047ddcb38b-tls-certs\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.074032 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.074006 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-home\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.074088 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.074014 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-model-cache\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.074088 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.074078 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-tmp-dir\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.075997 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.075977 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-dshm\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.076183 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.076165 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a85083-96ea-45e8-8574-fd047ddcb38b-tls-certs\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.082798 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.082744 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrdx\" (UniqueName: \"kubernetes.io/projected/25a85083-96ea-45e8-8574-fd047ddcb38b-kube-api-access-nfrdx\") pod \"test-model-c5c936-kserve-86cdc4b5ff-4ldzz\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.144913 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.144888 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:23.288446 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.288419 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz"] Jul 06 12:54:23.290879 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:54:23.290848 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25a85083_96ea_45e8_8574_fd047ddcb38b.slice/crio-08699f51a32c3940db9f00e68a5b12b264b12470b75693c44a3feba1b9b0a31e WatchSource:0}: Error finding container 08699f51a32c3940db9f00e68a5b12b264b12470b75693c44a3feba1b9b0a31e: Status 404 returned error can't find the container with id 08699f51a32c3940db9f00e68a5b12b264b12470b75693c44a3feba1b9b0a31e Jul 06 12:54:23.701160 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.701064 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" event={"ID":"25a85083-96ea-45e8-8574-fd047ddcb38b","Type":"ContainerStarted","Data":"eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b"} Jul 06 12:54:23.701160 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.701107 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" event={"ID":"25a85083-96ea-45e8-8574-fd047ddcb38b","Type":"ContainerStarted","Data":"08699f51a32c3940db9f00e68a5b12b264b12470b75693c44a3feba1b9b0a31e"} Jul 06 12:54:23.720485 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:23.720436 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" podStartSLOduration=1.720420619 podStartE2EDuration="1.720420619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:54:23.719487159 +0000 UTC m=+1879.798035187" watchObservedRunningTime="2026-07-06 12:54:23.720420619 +0000 UTC m=+1879.798968645" Jul 06 12:54:28.232145 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:28.232063 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:54:33.145635 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:33.145596 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:33.146095 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:33.145717 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:33.158865 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:33.158841 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:33.764349 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:33.764317 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:41.114623 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:41.114591 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:54:42.586511 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:42.586478 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2"] Jul 06 12:54:42.610365 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:42.610328 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:43.576048 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:43.576014 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz"] Jul 06 12:54:43.608580 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:43.608549 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:43.855902 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:43.855820 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:44.256306 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:44.256270 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:44.496997 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:44.496963 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:44.595965 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:44.595876 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:44.726611 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:44.726578 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:49.286187 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:49.286153 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:54:55.810530 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:55.810491 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:54:57.605997 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.605951 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" podUID="4c0afae9-3c5d-4210-aa74-743906b1fd22" containerName="main" containerID="cri-o://e2bae22f26377822ab26e0f214c7ad6b310bf00eec0b11f119df39d008b769b2" gracePeriod=45 Jul 06 12:54:57.855031 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.854998 2572 generic.go:356] "Generic (PLEG): container finished" podID="4c0afae9-3c5d-4210-aa74-743906b1fd22" containerID="e2bae22f26377822ab26e0f214c7ad6b310bf00eec0b11f119df39d008b769b2" exitCode=0 Jul 06 12:54:57.855176 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.855072 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" event={"ID":"4c0afae9-3c5d-4210-aa74-743906b1fd22","Type":"ContainerDied","Data":"e2bae22f26377822ab26e0f214c7ad6b310bf00eec0b11f119df39d008b769b2"} Jul 06 12:54:57.855176 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.855113 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" event={"ID":"4c0afae9-3c5d-4210-aa74-743906b1fd22","Type":"ContainerDied","Data":"e205281603328789b27a3abb73ae11f2da3b4209fcbbebecb040f831ceeba241"} Jul 06 12:54:57.855176 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.855125 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e205281603328789b27a3abb73ae11f2da3b4209fcbbebecb040f831ceeba241" Jul 06 12:54:57.857644 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.857596 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:57.974788 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.974744 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c0afae9-3c5d-4210-aa74-743906b1fd22-tls-certs\") pod \"4c0afae9-3c5d-4210-aa74-743906b1fd22\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " Jul 06 12:54:57.974968 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.974848 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-home\") pod \"4c0afae9-3c5d-4210-aa74-743906b1fd22\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " Jul 06 12:54:57.974968 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.974870 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-dshm\") pod \"4c0afae9-3c5d-4210-aa74-743906b1fd22\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " Jul 06 12:54:57.974968 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.974891 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-model-cache\") pod \"4c0afae9-3c5d-4210-aa74-743906b1fd22\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " Jul 06 12:54:57.974968 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.974912 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-tmp-dir\") pod \"4c0afae9-3c5d-4210-aa74-743906b1fd22\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " Jul 06 12:54:57.974968 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.974942 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbvhm\" (UniqueName: \"kubernetes.io/projected/4c0afae9-3c5d-4210-aa74-743906b1fd22-kube-api-access-vbvhm\") pod \"4c0afae9-3c5d-4210-aa74-743906b1fd22\" (UID: \"4c0afae9-3c5d-4210-aa74-743906b1fd22\") " Jul 06 12:54:57.975239 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.975155 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c0afae9-3c5d-4210-aa74-743906b1fd22" (UID: "4c0afae9-3c5d-4210-aa74-743906b1fd22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:57.975239 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.975215 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-home" (OuterVolumeSpecName: "home") pod "4c0afae9-3c5d-4210-aa74-743906b1fd22" (UID: "4c0afae9-3c5d-4210-aa74-743906b1fd22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:57.975239 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.975233 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c0afae9-3c5d-4210-aa74-743906b1fd22" (UID: "4c0afae9-3c5d-4210-aa74-743906b1fd22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:57.977093 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.977066 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-dshm" (OuterVolumeSpecName: "dshm") pod "4c0afae9-3c5d-4210-aa74-743906b1fd22" (UID: "4c0afae9-3c5d-4210-aa74-743906b1fd22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:57.977093 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.977064 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0afae9-3c5d-4210-aa74-743906b1fd22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c0afae9-3c5d-4210-aa74-743906b1fd22" (UID: "4c0afae9-3c5d-4210-aa74-743906b1fd22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:54:57.977232 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:57.977063 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0afae9-3c5d-4210-aa74-743906b1fd22-kube-api-access-vbvhm" (OuterVolumeSpecName: "kube-api-access-vbvhm") pod "4c0afae9-3c5d-4210-aa74-743906b1fd22" (UID: "4c0afae9-3c5d-4210-aa74-743906b1fd22"). InnerVolumeSpecName "kube-api-access-vbvhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:54:58.075852 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.075827 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:58.075852 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.075847 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:58.075852 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.075856 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:58.076044 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.075865 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c0afae9-3c5d-4210-aa74-743906b1fd22-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:58.076044 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.075874 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvhm\" (UniqueName: \"kubernetes.io/projected/4c0afae9-3c5d-4210-aa74-743906b1fd22-kube-api-access-vbvhm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:58.076044 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.075885 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c0afae9-3c5d-4210-aa74-743906b1fd22-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:58.595408 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.595349 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" podUID="25a85083-96ea-45e8-8574-fd047ddcb38b" containerName="main" containerID="cri-o://eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b" gracePeriod=45 Jul 06 12:54:58.835427 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.835406 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:58.860829 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.860759 2572 generic.go:356] "Generic (PLEG): container finished" podID="25a85083-96ea-45e8-8574-fd047ddcb38b" containerID="eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b" exitCode=0 Jul 06 12:54:58.860829 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.860797 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" Jul 06 12:54:58.861019 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.860836 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" event={"ID":"25a85083-96ea-45e8-8574-fd047ddcb38b","Type":"ContainerDied","Data":"eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b"} Jul 06 12:54:58.861019 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.860873 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz" event={"ID":"25a85083-96ea-45e8-8574-fd047ddcb38b","Type":"ContainerDied","Data":"08699f51a32c3940db9f00e68a5b12b264b12470b75693c44a3feba1b9b0a31e"} Jul 06 12:54:58.861019 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.860890 2572 scope.go:117] "RemoveContainer" containerID="eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b" Jul 06 12:54:58.861019 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.860939 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2" Jul 06 12:54:58.870368 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.870353 2572 scope.go:117] "RemoveContainer" containerID="eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b" Jul 06 12:54:58.870730 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:54:58.870644 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b\": container with ID starting with eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b not found: ID does not exist" containerID="eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b" Jul 06 12:54:58.870730 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.870696 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b"} err="failed to get container status \"eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b\": rpc error: code = NotFound desc = could not find container \"eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b\": container with ID starting with eebf8667ed9beb7d487c7cc354504db7ec0476eb590bcc6b2a0851f93946c61b not found: ID does not exist" Jul 06 12:54:58.887576 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.887543 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2"] Jul 06 12:54:58.891355 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.891335 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-86ab8b/test-model-86ab8b-kserve-b6fd4d555-488b2"] Jul 06 12:54:58.984573 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.984546 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-home\") pod \"25a85083-96ea-45e8-8574-fd047ddcb38b\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " Jul 06 12:54:58.984734 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.984585 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-model-cache\") pod \"25a85083-96ea-45e8-8574-fd047ddcb38b\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " Jul 06 12:54:58.984734 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.984610 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-tmp-dir\") pod \"25a85083-96ea-45e8-8574-fd047ddcb38b\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " Jul 06 12:54:58.984734 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.984630 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfrdx\" (UniqueName: \"kubernetes.io/projected/25a85083-96ea-45e8-8574-fd047ddcb38b-kube-api-access-nfrdx\") pod \"25a85083-96ea-45e8-8574-fd047ddcb38b\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " Jul 06 12:54:58.984734 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.984703 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a85083-96ea-45e8-8574-fd047ddcb38b-tls-certs\") pod \"25a85083-96ea-45e8-8574-fd047ddcb38b\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " Jul 06 12:54:58.984958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.984757 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-dshm\") pod \"25a85083-96ea-45e8-8574-fd047ddcb38b\" (UID: \"25a85083-96ea-45e8-8574-fd047ddcb38b\") " Jul 06 12:54:58.984958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.984867 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-home" (OuterVolumeSpecName: "home") pod "25a85083-96ea-45e8-8574-fd047ddcb38b" (UID: "25a85083-96ea-45e8-8574-fd047ddcb38b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:58.984958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.984878 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-model-cache" (OuterVolumeSpecName: "model-cache") pod "25a85083-96ea-45e8-8574-fd047ddcb38b" (UID: "25a85083-96ea-45e8-8574-fd047ddcb38b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:58.984958 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.984891 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25a85083-96ea-45e8-8574-fd047ddcb38b" (UID: "25a85083-96ea-45e8-8574-fd047ddcb38b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:58.985150 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.985043 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:58.985150 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.985064 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:58.985150 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.985082 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:58.986810 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.986785 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a85083-96ea-45e8-8574-fd047ddcb38b-kube-api-access-nfrdx" (OuterVolumeSpecName: "kube-api-access-nfrdx") pod "25a85083-96ea-45e8-8574-fd047ddcb38b" (UID: "25a85083-96ea-45e8-8574-fd047ddcb38b"). InnerVolumeSpecName "kube-api-access-nfrdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:54:58.986910 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.986813 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25a85083-96ea-45e8-8574-fd047ddcb38b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25a85083-96ea-45e8-8574-fd047ddcb38b" (UID: "25a85083-96ea-45e8-8574-fd047ddcb38b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:54:58.986957 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:58.986917 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-dshm" (OuterVolumeSpecName: "dshm") pod "25a85083-96ea-45e8-8574-fd047ddcb38b" (UID: "25a85083-96ea-45e8-8574-fd047ddcb38b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:59.086216 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:59.086185 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a85083-96ea-45e8-8574-fd047ddcb38b-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:59.086216 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:59.086211 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a85083-96ea-45e8-8574-fd047ddcb38b-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:59.086372 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:59.086225 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrdx\" (UniqueName: \"kubernetes.io/projected/25a85083-96ea-45e8-8574-fd047ddcb38b-kube-api-access-nfrdx\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:54:59.183515 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:59.183488 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz"] Jul 06 12:54:59.188402 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:59.188376 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c5c936/test-model-c5c936-kserve-86cdc4b5ff-4ldzz"] Jul 06 12:54:59.830846 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:59.830809 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25a85083-96ea-45e8-8574-fd047ddcb38b" path="/var/lib/kubelet/pods/25a85083-96ea-45e8-8574-fd047ddcb38b/volumes" Jul 06 12:54:59.831275 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:54:59.831254 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c0afae9-3c5d-4210-aa74-743906b1fd22" path="/var/lib/kubelet/pods/4c0afae9-3c5d-4210-aa74-743906b1fd22/volumes" Jul 06 12:55:25.609824 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:25.609787 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:25.710571 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:25.710539 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:26.654468 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:26.654431 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:26.991452 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:26.991382 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:27.389168 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:27.389080 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:27.754324 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:27.754279 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:27.932164 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:27.932134 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:28.031049 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:28.030951 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:28.237382 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:28.237350 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:31.221765 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:31.221731 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:55:34.273300 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:34.273260 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:34.371401 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:34.371345 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:34.530639 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:34.530555 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:35.110374 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:35.110338 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:35.463398 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:35.463348 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:35.634271 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:35.634243 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:38.231521 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:38.231432 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:38.346188 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:38.346149 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:38.439589 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:38.439553 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:39.133145 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:39.133112 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:39.240057 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:39.240011 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:42.744292 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:42.744248 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:43.259949 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:43.259919 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:43.711984 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:43.711953 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:55:43.758243 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:43.758205 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:44.165175 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:44.165082 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:47.419225 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:47.419185 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:47.562527 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:47.562469 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:47.664932 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:47.664897 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:48.188574 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:48.188534 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:48.333171 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:48.333138 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:48.433798 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:48.433764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:49.750321 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:49.750285 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:49.852635 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:49.852588 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:50.261021 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:50.260971 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:51.888866 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:51.888835 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:55:59.504635 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:59.504558 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:55:59.990263 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:55:59.990234 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:00.408129 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:00.408044 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:00.684848 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:00.684750 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:00.785309 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:00.785279 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:00.974008 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:00.973968 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:06.802373 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:06.802336 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:06.906502 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:06.906467 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:07.101436 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:07.101338 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:07.531287 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:07.531238 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:07.839044 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:07.838957 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:07.943814 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:07.943781 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:08.040871 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:08.040836 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:08.239761 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:08.239725 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:11.514537 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:11.514494 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:56:14.116913 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:14.116877 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:14.287630 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:14.287575 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:14.501777 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:14.501747 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:14.971397 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:14.971365 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:15.269380 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:15.269293 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:15.421631 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:15.421600 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:18.597924 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:18.597890 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:18.716274 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:18.716244 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:18.809733 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:18.809694 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:19.303160 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:19.303122 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:19.400049 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:19.399996 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:22.955117 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:22.955084 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:23.281906 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:23.281871 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:23.624044 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:23.623965 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:23.812868 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:23.812840 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:26.109462 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:26.109426 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:56:26.622040 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:26.622003 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:26.718505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:26.718473 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:26.887259 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:26.887177 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:27.235772 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:27.235736 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:27.629024 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:27.628932 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:27.728948 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:27.728901 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:31.115165 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:31.115130 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:40.097142 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:40.097100 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:40.195137 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:40.195107 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:40.204243 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:40.204220 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:56:49.784957 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:49.784915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:49.893798 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:49.893764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:51.238481 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.238445 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv"] Jul 06 12:56:51.238863 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.238787 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a85083-96ea-45e8-8574-fd047ddcb38b" containerName="main" Jul 06 12:56:51.238863 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.238799 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a85083-96ea-45e8-8574-fd047ddcb38b" containerName="main" Jul 06 12:56:51.238863 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.238812 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0afae9-3c5d-4210-aa74-743906b1fd22" containerName="main" Jul 06 12:56:51.238863 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.238818 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0afae9-3c5d-4210-aa74-743906b1fd22" containerName="main" Jul 06 12:56:51.239031 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.238876 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="25a85083-96ea-45e8-8574-fd047ddcb38b" containerName="main" Jul 06 12:56:51.239031 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.238888 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0afae9-3c5d-4210-aa74-743906b1fd22" containerName="main" Jul 06 12:56:51.241807 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.241791 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.247345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.247310 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e47695\"/\"openshift-service-ca.crt\"" Jul 06 12:56:51.247345 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.247328 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e47695\"/\"auth-test-model-e47695-kserve-self-signed-certs\"" Jul 06 12:56:51.247526 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.247319 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e47695\"/\"default-dockercfg-6sxfw\"" Jul 06 12:56:51.247526 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.247373 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e47695\"/\"kube-root-ca.crt\"" Jul 06 12:56:51.254056 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.254034 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv"] Jul 06 12:56:51.277810 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.277787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23100c8e-3c45-405c-b2fb-8171b9edea7a-tls-certs\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.277931 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.277825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-home\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.277931 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.277889 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-model-cache\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.278096 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.277934 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-tmp-dir\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.278096 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.277982 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddhv\" (UniqueName: \"kubernetes.io/projected/23100c8e-3c45-405c-b2fb-8171b9edea7a-kube-api-access-nddhv\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.278096 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.278032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-dshm\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.279434 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.279407 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:56:51.378766 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.378743 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23100c8e-3c45-405c-b2fb-8171b9edea7a-tls-certs\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.378916 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.378781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-home\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.378916 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.378813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-model-cache\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.378916 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.378843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-tmp-dir\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.378916 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.378879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddhv\" (UniqueName: \"kubernetes.io/projected/23100c8e-3c45-405c-b2fb-8171b9edea7a-kube-api-access-nddhv\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.379140 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.378921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-dshm\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.379197 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.379175 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-home\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.379269 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.379242 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-model-cache\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.379324 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.379258 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-tmp-dir\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.381133 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.381112 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-dshm\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.381371 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.381356 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23100c8e-3c45-405c-b2fb-8171b9edea7a-tls-certs\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.388199 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.388178 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddhv\" (UniqueName: \"kubernetes.io/projected/23100c8e-3c45-405c-b2fb-8171b9edea7a-kube-api-access-nddhv\") pod \"auth-test-model-e47695-kserve-d7b8df6c-gfdbv\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.550738 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.550644 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:56:51.685227 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:51.685197 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv"] Jul 06 12:56:51.687774 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:56:51.687743 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23100c8e_3c45_405c_b2fb_8171b9edea7a.slice/crio-71b59b9289bb8882303d24dd5d72c0658d20254a38154c62a208afc487a6f502 WatchSource:0}: Error finding container 71b59b9289bb8882303d24dd5d72c0658d20254a38154c62a208afc487a6f502: Status 404 returned error can't find the container with id 71b59b9289bb8882303d24dd5d72c0658d20254a38154c62a208afc487a6f502 Jul 06 12:56:52.329578 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:52.329539 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" event={"ID":"23100c8e-3c45-405c-b2fb-8171b9edea7a","Type":"ContainerStarted","Data":"3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3"} Jul 06 12:56:52.329985 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:52.329587 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" event={"ID":"23100c8e-3c45-405c-b2fb-8171b9edea7a","Type":"ContainerStarted","Data":"71b59b9289bb8882303d24dd5d72c0658d20254a38154c62a208afc487a6f502"} Jul 06 12:56:52.349851 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:52.349773 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" podStartSLOduration=1.349757398 podStartE2EDuration="1.349757398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:56:52.349056193 +0000 UTC m=+2028.427604220" watchObservedRunningTime="2026-07-06 12:56:52.349757398 +0000 UTC m=+2028.428305426" Jul 06 12:56:54.304204 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:56:54.304164 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:57:01.551410 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:01.551360 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:57:01.551917 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:01.551453 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:57:01.563993 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:01.563968 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:57:02.380374 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:02.380349 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:57:13.330213 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.330172 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b"] Jul 06 12:57:13.333393 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.333371 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.336150 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.336129 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"kube-root-ca.crt\"" Jul 06 12:57:13.337370 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.337349 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"default-dockercfg-phhqd\"" Jul 06 12:57:13.337468 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.337406 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"openshift-service-ca.crt\"" Jul 06 12:57:13.337468 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.337439 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"auth-test-model-984f35-kserve-self-signed-certs\"" Jul 06 12:57:13.348067 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.348041 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b"] Jul 06 12:57:13.352247 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.352212 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:57:13.462209 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.462170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-home\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.462209 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.462211 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-dshm\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.462443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.462248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-model-cache\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.462443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.462313 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-tmp-dir\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.462443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.462358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqckl\" (UniqueName: \"kubernetes.io/projected/83435c02-394b-4691-ab8a-d2243802a692-kube-api-access-kqckl\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.462443 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.462390 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83435c02-394b-4691-ab8a-d2243802a692-tls-certs\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.563038 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.563000 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-tmp-dir\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.563188 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.563045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqckl\" (UniqueName: \"kubernetes.io/projected/83435c02-394b-4691-ab8a-d2243802a692-kube-api-access-kqckl\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.563188 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.563073 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83435c02-394b-4691-ab8a-d2243802a692-tls-certs\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.563188 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.563095 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-home\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.563188 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.563113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-dshm\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.563188 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.563140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-model-cache\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.563490 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.563466 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-tmp-dir\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.563561 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.563489 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-model-cache\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.563637 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.563617 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-home\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.565401 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.565379 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-dshm\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.565538 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.565521 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83435c02-394b-4691-ab8a-d2243802a692-tls-certs\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.573485 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.573463 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqckl\" (UniqueName: \"kubernetes.io/projected/83435c02-394b-4691-ab8a-d2243802a692-kube-api-access-kqckl\") pod \"auth-test-model-984f35-kserve-57d6847495-pn44b\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.642592 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.642500 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:13.776422 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:13.776397 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b"] Jul 06 12:57:13.778374 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:57:13.778343 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83435c02_394b_4691_ab8a_d2243802a692.slice/crio-361b7b084257c2ad269c5fbdc31e399bdc7ec2b669ef05870d490eaf6c2f8c95 WatchSource:0}: Error finding container 361b7b084257c2ad269c5fbdc31e399bdc7ec2b669ef05870d490eaf6c2f8c95: Status 404 returned error can't find the container with id 361b7b084257c2ad269c5fbdc31e399bdc7ec2b669ef05870d490eaf6c2f8c95 Jul 06 12:57:14.420112 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:14.420071 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" event={"ID":"83435c02-394b-4691-ab8a-d2243802a692","Type":"ContainerStarted","Data":"f7ef6a6ad88cf5904f33df39bbb997ea4cfff72c9e66194507152979206deadd"} Jul 06 12:57:14.420528 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:14.420123 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" event={"ID":"83435c02-394b-4691-ab8a-d2243802a692","Type":"ContainerStarted","Data":"361b7b084257c2ad269c5fbdc31e399bdc7ec2b669ef05870d490eaf6c2f8c95"} Jul 06 12:57:14.440111 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:14.440059 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" podStartSLOduration=1.440041855 podStartE2EDuration="1.440041855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:57:14.438701302 +0000 UTC m=+2050.517249327" watchObservedRunningTime="2026-07-06 12:57:14.440041855 +0000 UTC m=+2050.518589882" Jul 06 12:57:16.858051 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:16.858015 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:57:23.642814 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:23.642760 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:23.642814 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:23.642822 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:23.655254 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:23.655226 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:24.473225 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:24.473194 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:57:34.804919 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:34.804886 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:57:48.519211 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:57:48.519174 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:58:04.309713 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:58:04.309679 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:58:19.824947 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:58:19.824917 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:58:28.937941 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:58:28.937898 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:58:30.121188 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:58:30.121151 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:58:35.512284 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:58:35.512247 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:58:52.312428 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:58:52.312390 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:59:02.949921 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:02.949845 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv"] Jul 06 12:59:03.014563 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:03.014535 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:59:03.758499 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:03.758467 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b"] Jul 06 12:59:03.818519 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:03.818486 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:59:07.311394 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:07.311351 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:59:11.786923 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:11.786888 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:59:17.969144 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:17.969101 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" podUID="23100c8e-3c45-405c-b2fb-8171b9edea7a" containerName="main" containerID="cri-o://3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3" gracePeriod=45 Jul 06 12:59:18.208905 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.208878 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:59:18.351748 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.351631 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-model-cache\") pod \"23100c8e-3c45-405c-b2fb-8171b9edea7a\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " Jul 06 12:59:18.351748 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.351728 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-dshm\") pod \"23100c8e-3c45-405c-b2fb-8171b9edea7a\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " Jul 06 12:59:18.351975 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.351763 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23100c8e-3c45-405c-b2fb-8171b9edea7a-tls-certs\") pod \"23100c8e-3c45-405c-b2fb-8171b9edea7a\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " Jul 06 12:59:18.351975 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.351822 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-home\") pod \"23100c8e-3c45-405c-b2fb-8171b9edea7a\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " Jul 06 12:59:18.351975 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.351852 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-tmp-dir\") pod \"23100c8e-3c45-405c-b2fb-8171b9edea7a\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " Jul 06 12:59:18.351975 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.351884 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nddhv\" (UniqueName: \"kubernetes.io/projected/23100c8e-3c45-405c-b2fb-8171b9edea7a-kube-api-access-nddhv\") pod \"23100c8e-3c45-405c-b2fb-8171b9edea7a\" (UID: \"23100c8e-3c45-405c-b2fb-8171b9edea7a\") " Jul 06 12:59:18.351975 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.351891 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "23100c8e-3c45-405c-b2fb-8171b9edea7a" (UID: "23100c8e-3c45-405c-b2fb-8171b9edea7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:59:18.352222 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.352087 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-home" (OuterVolumeSpecName: "home") pod "23100c8e-3c45-405c-b2fb-8171b9edea7a" (UID: "23100c8e-3c45-405c-b2fb-8171b9edea7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:59:18.352222 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.352104 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23100c8e-3c45-405c-b2fb-8171b9edea7a" (UID: "23100c8e-3c45-405c-b2fb-8171b9edea7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:59:18.352222 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.352176 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:18.354054 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.354029 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23100c8e-3c45-405c-b2fb-8171b9edea7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23100c8e-3c45-405c-b2fb-8171b9edea7a" (UID: "23100c8e-3c45-405c-b2fb-8171b9edea7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:59:18.354139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.354029 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23100c8e-3c45-405c-b2fb-8171b9edea7a-kube-api-access-nddhv" (OuterVolumeSpecName: "kube-api-access-nddhv") pod "23100c8e-3c45-405c-b2fb-8171b9edea7a" (UID: "23100c8e-3c45-405c-b2fb-8171b9edea7a"). InnerVolumeSpecName "kube-api-access-nddhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:59:18.354139 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.354035 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-dshm" (OuterVolumeSpecName: "dshm") pod "23100c8e-3c45-405c-b2fb-8171b9edea7a" (UID: "23100c8e-3c45-405c-b2fb-8171b9edea7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:59:18.453505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.453479 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:18.453505 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.453505 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23100c8e-3c45-405c-b2fb-8171b9edea7a-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:18.453694 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.453515 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:18.453694 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.453523 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23100c8e-3c45-405c-b2fb-8171b9edea7a-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:18.453694 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.453531 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddhv\" (UniqueName: \"kubernetes.io/projected/23100c8e-3c45-405c-b2fb-8171b9edea7a-kube-api-access-nddhv\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:18.778379 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.778323 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" podUID="83435c02-394b-4691-ab8a-d2243802a692" containerName="main" containerID="cri-o://f7ef6a6ad88cf5904f33df39bbb997ea4cfff72c9e66194507152979206deadd" gracePeriod=45 Jul 06 12:59:18.952143 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.952115 2572 generic.go:356] "Generic (PLEG): container finished" podID="83435c02-394b-4691-ab8a-d2243802a692" containerID="f7ef6a6ad88cf5904f33df39bbb997ea4cfff72c9e66194507152979206deadd" exitCode=0 Jul 06 12:59:18.952268 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.952183 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" event={"ID":"83435c02-394b-4691-ab8a-d2243802a692","Type":"ContainerDied","Data":"f7ef6a6ad88cf5904f33df39bbb997ea4cfff72c9e66194507152979206deadd"} Jul 06 12:59:18.953605 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.953580 2572 generic.go:356] "Generic (PLEG): container finished" podID="23100c8e-3c45-405c-b2fb-8171b9edea7a" containerID="3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3" exitCode=0 Jul 06 12:59:18.953730 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.953618 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" event={"ID":"23100c8e-3c45-405c-b2fb-8171b9edea7a","Type":"ContainerDied","Data":"3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3"} Jul 06 12:59:18.953730 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.953632 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" Jul 06 12:59:18.953730 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.953648 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv" event={"ID":"23100c8e-3c45-405c-b2fb-8171b9edea7a","Type":"ContainerDied","Data":"71b59b9289bb8882303d24dd5d72c0658d20254a38154c62a208afc487a6f502"} Jul 06 12:59:18.953730 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.953686 2572 scope.go:117] "RemoveContainer" containerID="3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3" Jul 06 12:59:18.961790 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.961764 2572 scope.go:117] "RemoveContainer" containerID="3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3" Jul 06 12:59:18.962071 ip-10-0-136-228 kubenswrapper[2572]: E0706 12:59:18.962049 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3\": container with ID starting with 3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3 not found: ID does not exist" containerID="3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3" Jul 06 12:59:18.962152 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.962079 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3"} err="failed to get container status \"3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3\": rpc error: code = NotFound desc = could not find container \"3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3\": container with ID starting with 3ce8612982661c3f45939245f517bb6bbbaa7041e585af33b2aad59266a25da3 not found: ID does not exist" Jul 06 12:59:18.981114 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.981093 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv"] Jul 06 12:59:18.984091 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:18.984069 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/auth-test-model-e47695-kserve-d7b8df6c-gfdbv"] Jul 06 12:59:19.031438 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.031385 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:59:19.158678 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.158620 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqckl\" (UniqueName: \"kubernetes.io/projected/83435c02-394b-4691-ab8a-d2243802a692-kube-api-access-kqckl\") pod \"83435c02-394b-4691-ab8a-d2243802a692\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " Jul 06 12:59:19.158867 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.158687 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-home\") pod \"83435c02-394b-4691-ab8a-d2243802a692\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " Jul 06 12:59:19.158867 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.158755 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83435c02-394b-4691-ab8a-d2243802a692-tls-certs\") pod \"83435c02-394b-4691-ab8a-d2243802a692\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " Jul 06 12:59:19.158867 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.158781 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-model-cache\") pod \"83435c02-394b-4691-ab8a-d2243802a692\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " Jul 06 12:59:19.159041 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.158959 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-tmp-dir\") pod \"83435c02-394b-4691-ab8a-d2243802a692\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " Jul 06 12:59:19.159041 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.158974 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-home" (OuterVolumeSpecName: "home") pod "83435c02-394b-4691-ab8a-d2243802a692" (UID: "83435c02-394b-4691-ab8a-d2243802a692"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:59:19.159041 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.158996 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-model-cache" (OuterVolumeSpecName: "model-cache") pod "83435c02-394b-4691-ab8a-d2243802a692" (UID: "83435c02-394b-4691-ab8a-d2243802a692"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:59:19.159163 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.159039 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-dshm\") pod \"83435c02-394b-4691-ab8a-d2243802a692\" (UID: \"83435c02-394b-4691-ab8a-d2243802a692\") " Jul 06 12:59:19.159224 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.159205 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83435c02-394b-4691-ab8a-d2243802a692" (UID: "83435c02-394b-4691-ab8a-d2243802a692"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:59:19.159388 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.159369 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:19.159456 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.159394 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:19.159456 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.159407 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:19.160967 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.160944 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-dshm" (OuterVolumeSpecName: "dshm") pod "83435c02-394b-4691-ab8a-d2243802a692" (UID: "83435c02-394b-4691-ab8a-d2243802a692"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:59:19.161066 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.160993 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83435c02-394b-4691-ab8a-d2243802a692-kube-api-access-kqckl" (OuterVolumeSpecName: "kube-api-access-kqckl") pod "83435c02-394b-4691-ab8a-d2243802a692" (UID: "83435c02-394b-4691-ab8a-d2243802a692"). InnerVolumeSpecName "kube-api-access-kqckl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:59:19.161115 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.161067 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83435c02-394b-4691-ab8a-d2243802a692-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83435c02-394b-4691-ab8a-d2243802a692" (UID: "83435c02-394b-4691-ab8a-d2243802a692"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:59:19.259805 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.259761 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83435c02-394b-4691-ab8a-d2243802a692-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:19.259805 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.259799 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83435c02-394b-4691-ab8a-d2243802a692-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:19.259805 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.259811 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqckl\" (UniqueName: \"kubernetes.io/projected/83435c02-394b-4691-ab8a-d2243802a692-kube-api-access-kqckl\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 12:59:19.831192 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.831163 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23100c8e-3c45-405c-b2fb-8171b9edea7a" path="/var/lib/kubelet/pods/23100c8e-3c45-405c-b2fb-8171b9edea7a/volumes" Jul 06 12:59:19.958871 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.958843 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" Jul 06 12:59:19.958871 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.958866 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b" event={"ID":"83435c02-394b-4691-ab8a-d2243802a692","Type":"ContainerDied","Data":"361b7b084257c2ad269c5fbdc31e399bdc7ec2b669ef05870d490eaf6c2f8c95"} Jul 06 12:59:19.959105 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.958921 2572 scope.go:117] "RemoveContainer" containerID="f7ef6a6ad88cf5904f33df39bbb997ea4cfff72c9e66194507152979206deadd" Jul 06 12:59:19.976608 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.976582 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b"] Jul 06 12:59:19.982376 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:19.982355 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/auth-test-model-984f35-kserve-57d6847495-pn44b"] Jul 06 12:59:21.313448 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:21.313406 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:59:21.831180 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:21.831148 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83435c02-394b-4691-ab8a-d2243802a692" path="/var/lib/kubelet/pods/83435c02-394b-4691-ab8a-d2243802a692/volumes" Jul 06 12:59:46.743233 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.743200 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl"] Jul 06 12:59:46.743742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.743495 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83435c02-394b-4691-ab8a-d2243802a692" containerName="main" Jul 06 12:59:46.743742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.743505 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83435c02-394b-4691-ab8a-d2243802a692" containerName="main" Jul 06 12:59:46.743742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.743515 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23100c8e-3c45-405c-b2fb-8171b9edea7a" containerName="main" Jul 06 12:59:46.743742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.743520 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="23100c8e-3c45-405c-b2fb-8171b9edea7a" containerName="main" Jul 06 12:59:46.743742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.743586 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="23100c8e-3c45-405c-b2fb-8171b9edea7a" containerName="main" Jul 06 12:59:46.743742 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.743596 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="83435c02-394b-4691-ab8a-d2243802a692" containerName="main" Jul 06 12:59:46.746419 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.746403 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.749340 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.749319 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"openshift-service-ca.crt\"" Jul 06 12:59:46.750582 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.750555 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"default-dockercfg-phhqd\"" Jul 06 12:59:46.750723 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.750625 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"kube-root-ca.crt\"" Jul 06 12:59:46.750796 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.750788 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"sub-test-model-984f35-kserve-self-signed-certs\"" Jul 06 12:59:46.758584 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.758560 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl"] Jul 06 12:59:46.779435 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.779410 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:59:46.885935 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.885908 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tls-certs\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.886080 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.885959 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdthp\" (UniqueName: \"kubernetes.io/projected/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-kube-api-access-gdthp\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.886080 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.885979 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-dshm\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.886080 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.886053 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-model-cache\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.886220 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.886097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-home\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.886220 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.886148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tmp-dir\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.986860 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.986831 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tls-certs\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.987004 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.986891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdthp\" (UniqueName: \"kubernetes.io/projected/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-kube-api-access-gdthp\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.987004 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.986919 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-dshm\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.987004 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.986963 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-model-cache\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.987151 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.987091 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-home\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.987217 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.987156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tmp-dir\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.987407 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.987388 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-model-cache\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.987489 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.987465 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-home\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.987540 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.987487 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tmp-dir\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.989197 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.989173 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-dshm\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.989401 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.989385 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tls-certs\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:46.995739 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:46.995639 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdthp\" (UniqueName: \"kubernetes.io/projected/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-kube-api-access-gdthp\") pod \"sub-test-model-984f35-kserve-75d475878f-lw2hl\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:47.055558 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:47.055534 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:47.191642 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:47.191615 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl"] Jul 06 12:59:47.194267 ip-10-0-136-228 kubenswrapper[2572]: W0706 12:59:47.194240 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f5dee6a_c5a5_4bb0_a436_6337a89ce1b0.slice/crio-c04b33f1042c8d0f9589df4ee34f2783a95e984dfb81b1777777a5d382061f3a WatchSource:0}: Error finding container c04b33f1042c8d0f9589df4ee34f2783a95e984dfb81b1777777a5d382061f3a: Status 404 returned error can't find the container with id c04b33f1042c8d0f9589df4ee34f2783a95e984dfb81b1777777a5d382061f3a Jul 06 12:59:48.082865 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:48.082827 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" event={"ID":"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0","Type":"ContainerStarted","Data":"d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f"} Jul 06 12:59:48.083232 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:48.082875 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" event={"ID":"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0","Type":"ContainerStarted","Data":"c04b33f1042c8d0f9589df4ee34f2783a95e984dfb81b1777777a5d382061f3a"} Jul 06 12:59:48.108652 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:48.108606 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" podStartSLOduration=2.108589244 podStartE2EDuration="2.108589244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:59:48.108235002 +0000 UTC m=+2204.186783029" watchObservedRunningTime="2026-07-06 12:59:48.108589244 +0000 UTC m=+2204.187137271" Jul 06 12:59:50.557147 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:50.557113 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 12:59:51.009259 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:51.009223 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 12:59:57.056223 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:57.056192 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:57.056699 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:57.056232 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:57.068151 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:57.068130 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 12:59:57.131030 ip-10-0-136-228 kubenswrapper[2572]: I0706 12:59:57.131008 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 13:00:00.154254 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.154215 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr"] Jul 06 13:00:00.158094 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.158070 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" Jul 06 13:00:00.162862 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.162838 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-frggv\"" Jul 06 13:00:00.163200 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.163182 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 06 13:00:00.163954 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.163935 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 06 13:00:00.171651 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.171628 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr"] Jul 06 13:00:00.302887 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.302851 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ph7\" (UniqueName: \"kubernetes.io/projected/ac6dd1af-5aed-4b69-ae98-dc290370f6dc-kube-api-access-w7ph7\") pod \"maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr\" (UID: \"ac6dd1af-5aed-4b69-ae98-dc290370f6dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" Jul 06 13:00:00.403529 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.403490 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ph7\" (UniqueName: \"kubernetes.io/projected/ac6dd1af-5aed-4b69-ae98-dc290370f6dc-kube-api-access-w7ph7\") pod \"maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr\" (UID: \"ac6dd1af-5aed-4b69-ae98-dc290370f6dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" Jul 06 13:00:00.413132 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.413061 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ph7\" (UniqueName: \"kubernetes.io/projected/ac6dd1af-5aed-4b69-ae98-dc290370f6dc-kube-api-access-w7ph7\") pod \"maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr\" (UID: \"ac6dd1af-5aed-4b69-ae98-dc290370f6dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" Jul 06 13:00:00.468292 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.468265 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" Jul 06 13:00:00.599628 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.599603 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr"] Jul 06 13:00:00.602175 ip-10-0-136-228 kubenswrapper[2572]: W0706 13:00:00.602141 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6dd1af_5aed_4b69_ae98_dc290370f6dc.slice/crio-d5d8afc4e8c5ff8ae85251b1f3b9cc841ca71e8c5bc7c720c2922ea7540a54b4 WatchSource:0}: Error finding container d5d8afc4e8c5ff8ae85251b1f3b9cc841ca71e8c5bc7c720c2922ea7540a54b4: Status 404 returned error can't find the container with id d5d8afc4e8c5ff8ae85251b1f3b9cc841ca71e8c5bc7c720c2922ea7540a54b4 Jul 06 13:00:00.604287 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:00.604269 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 13:00:01.140188 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:01.140148 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" event={"ID":"ac6dd1af-5aed-4b69-ae98-dc290370f6dc","Type":"ContainerStarted","Data":"d5d8afc4e8c5ff8ae85251b1f3b9cc841ca71e8c5bc7c720c2922ea7540a54b4"} Jul 06 13:00:05.931766 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:05.931734 2572 scope.go:117] "RemoveContainer" containerID="e2bae22f26377822ab26e0f214c7ad6b310bf00eec0b11f119df39d008b769b2" Jul 06 13:00:09.509885 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:09.509850 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 13:00:15.207535 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:15.207495 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" event={"ID":"ac6dd1af-5aed-4b69-ae98-dc290370f6dc","Type":"ContainerStarted","Data":"c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864"} Jul 06 13:00:15.235945 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:15.235901 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" podStartSLOduration=1.176640041 podStartE2EDuration="15.235885399s" totalImagesPullingTime="14.059245358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:00:15.234701133 +0000 UTC m=+2231.313249165" watchObservedRunningTime="2026-07-06 13:00:15.235885399 +0000 UTC m=+2231.314433425" Jul 06 13:00:23.913414 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:23.913379 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 13:00:28.861798 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:28.861703 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:00:29.809830 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:29.809790 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl"] Jul 06 13:00:29.841963 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:29.841924 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:00:39.514455 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:39.514420 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 13:00:44.829026 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:44.828986 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" podUID="7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" containerName="main" containerID="cri-o://d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f" gracePeriod=45 Jul 06 13:00:45.065377 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.065354 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 13:00:45.191801 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.191727 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-model-cache\") pod \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " Jul 06 13:00:45.191801 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.191767 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-dshm\") pod \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " Jul 06 13:00:45.192018 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.191831 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tls-certs\") pod \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " Jul 06 13:00:45.192018 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.191853 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdthp\" (UniqueName: \"kubernetes.io/projected/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-kube-api-access-gdthp\") pod \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " Jul 06 13:00:45.192018 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.191898 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-home\") pod \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " Jul 06 13:00:45.192018 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.191938 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tmp-dir\") pod \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\" (UID: \"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0\") " Jul 06 13:00:45.192018 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.191989 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" (UID: "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:00:45.192274 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.192201 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-home" (OuterVolumeSpecName: "home") pod "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" (UID: "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:00:45.192274 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.192231 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:00:45.192274 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.192212 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" (UID: "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:00:45.193925 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.193902 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-dshm" (OuterVolumeSpecName: "dshm") pod "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" (UID: "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:00:45.194039 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.193939 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-kube-api-access-gdthp" (OuterVolumeSpecName: "kube-api-access-gdthp") pod "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" (UID: "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0"). InnerVolumeSpecName "kube-api-access-gdthp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:00:45.194039 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.193972 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" (UID: "7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:00:45.293539 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.293508 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:00:45.293539 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.293533 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdthp\" (UniqueName: \"kubernetes.io/projected/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-kube-api-access-gdthp\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:00:45.293539 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.293541 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:00:45.293778 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.293550 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:00:45.293778 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.293559 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:00:45.332551 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.332521 2572 generic.go:356] "Generic (PLEG): container finished" podID="7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" containerID="d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f" exitCode=0 Jul 06 13:00:45.332680 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.332562 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" event={"ID":"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0","Type":"ContainerDied","Data":"d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f"} Jul 06 13:00:45.332680 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.332572 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" Jul 06 13:00:45.332680 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.332595 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl" event={"ID":"7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0","Type":"ContainerDied","Data":"c04b33f1042c8d0f9589df4ee34f2783a95e984dfb81b1777777a5d382061f3a"} Jul 06 13:00:45.332680 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.332617 2572 scope.go:117] "RemoveContainer" containerID="d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f" Jul 06 13:00:45.341222 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.341203 2572 scope.go:117] "RemoveContainer" containerID="d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f" Jul 06 13:00:45.341453 ip-10-0-136-228 kubenswrapper[2572]: E0706 13:00:45.341437 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f\": container with ID starting with d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f not found: ID does not exist" containerID="d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f" Jul 06 13:00:45.341505 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.341460 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f"} err="failed to get container status \"d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f\": rpc error: code = NotFound desc = could not find container \"d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f\": container with ID starting with d8cc1b7004b4d88d15bc52b4a05c1d871d2e325730a0b13997caaef99b4dee7f not found: ID does not exist" Jul 06 13:00:45.358150 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.358124 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl"] Jul 06 13:00:45.369574 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.369543 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/sub-test-model-984f35-kserve-75d475878f-lw2hl"] Jul 06 13:00:45.831459 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:45.831426 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" path="/var/lib/kubelet/pods/7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0/volumes" Jul 06 13:00:46.859934 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.859900 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp"] Jul 06 13:00:46.860299 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.860240 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" containerName="main" Jul 06 13:00:46.860299 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.860250 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" containerName="main" Jul 06 13:00:46.860369 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.860300 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f5dee6a-c5a5-4bb0-a436-6337a89ce1b0" containerName="main" Jul 06 13:00:46.867253 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.867235 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:46.874454 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.874436 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e47695\"/\"rate-test-model-e47695-kserve-self-signed-certs\"" Jul 06 13:00:46.874553 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.874488 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e47695\"/\"openshift-service-ca.crt\"" Jul 06 13:00:46.874605 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.874549 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e47695\"/\"default-dockercfg-6sxfw\"" Jul 06 13:00:46.874649 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.874616 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e47695\"/\"kube-root-ca.crt\"" Jul 06 13:00:46.885885 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:46.885859 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp"] Jul 06 13:00:47.006488 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.006459 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c51d897-9370-4f19-ae80-edf5d233748b-tls-certs\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.006648 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.006503 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-home\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.006648 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.006525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-dshm\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.006648 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.006602 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgd7d\" (UniqueName: \"kubernetes.io/projected/9c51d897-9370-4f19-ae80-edf5d233748b-kube-api-access-rgd7d\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.006828 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.006653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-model-cache\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.006828 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.006692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-tmp-dir\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.030899 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.030874 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:00:47.107496 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.107464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgd7d\" (UniqueName: \"kubernetes.io/projected/9c51d897-9370-4f19-ae80-edf5d233748b-kube-api-access-rgd7d\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.107651 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.107521 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-model-cache\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.107651 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.107555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-tmp-dir\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.107651 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.107599 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c51d897-9370-4f19-ae80-edf5d233748b-tls-certs\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.107838 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.107653 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-home\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.107838 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.107699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-dshm\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.108033 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.108007 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-tmp-dir\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.108104 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.108016 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-model-cache\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.108104 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.108066 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-home\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.109932 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.109913 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-dshm\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.110141 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.110103 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c51d897-9370-4f19-ae80-edf5d233748b-tls-certs\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.122757 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.122733 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgd7d\" (UniqueName: \"kubernetes.io/projected/9c51d897-9370-4f19-ae80-edf5d233748b-kube-api-access-rgd7d\") pod \"rate-test-model-e47695-kserve-86ddd8c79f-chwgp\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.175964 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.175938 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:47.324718 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.324637 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp"] Jul 06 13:00:47.327398 ip-10-0-136-228 kubenswrapper[2572]: W0706 13:00:47.327366 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c51d897_9370_4f19_ae80_edf5d233748b.slice/crio-c0319f615f8a6760e402f9d583c92887caef4d2eb3cb7db294daeb811c8869a0 WatchSource:0}: Error finding container c0319f615f8a6760e402f9d583c92887caef4d2eb3cb7db294daeb811c8869a0: Status 404 returned error can't find the container with id c0319f615f8a6760e402f9d583c92887caef4d2eb3cb7db294daeb811c8869a0 Jul 06 13:00:47.344949 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:47.344923 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" event={"ID":"9c51d897-9370-4f19-ae80-edf5d233748b","Type":"ContainerStarted","Data":"c0319f615f8a6760e402f9d583c92887caef4d2eb3cb7db294daeb811c8869a0"} Jul 06 13:00:48.350893 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:48.350858 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" event={"ID":"9c51d897-9370-4f19-ae80-edf5d233748b","Type":"ContainerStarted","Data":"7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a"} Jul 06 13:00:48.378970 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:48.378927 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" podStartSLOduration=2.378910667 podStartE2EDuration="2.378910667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:00:48.377373143 +0000 UTC m=+2264.455921170" watchObservedRunningTime="2026-07-06 13:00:48.378910667 +0000 UTC m=+2264.457458697" Jul 06 13:00:50.518033 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:50.517999 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 13:00:50.790863 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:50.790792 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:00:50.901550 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:50.901509 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:00:57.176074 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:57.176041 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:57.176480 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:57.176092 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:57.187893 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:57.187867 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:00:57.401389 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:00:57.401358 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:01:00.017053 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:00.017013 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr"] Jul 06 13:01:00.017615 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:00.017265 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" podUID="ac6dd1af-5aed-4b69-ae98-dc290370f6dc" containerName="cleanup" containerID="cri-o://c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864" gracePeriod=30 Jul 06 13:01:08.094863 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:08.094831 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:01:30.199521 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.199494 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" Jul 06 13:01:30.268280 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.268251 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7ph7\" (UniqueName: \"kubernetes.io/projected/ac6dd1af-5aed-4b69-ae98-dc290370f6dc-kube-api-access-w7ph7\") pod \"ac6dd1af-5aed-4b69-ae98-dc290370f6dc\" (UID: \"ac6dd1af-5aed-4b69-ae98-dc290370f6dc\") " Jul 06 13:01:30.270357 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.270324 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6dd1af-5aed-4b69-ae98-dc290370f6dc-kube-api-access-w7ph7" (OuterVolumeSpecName: "kube-api-access-w7ph7") pod "ac6dd1af-5aed-4b69-ae98-dc290370f6dc" (UID: "ac6dd1af-5aed-4b69-ae98-dc290370f6dc"). InnerVolumeSpecName "kube-api-access-w7ph7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:01:30.369116 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.369082 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7ph7\" (UniqueName: \"kubernetes.io/projected/ac6dd1af-5aed-4b69-ae98-dc290370f6dc-kube-api-access-w7ph7\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:01:30.529147 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.529073 2572 generic.go:356] "Generic (PLEG): container finished" podID="ac6dd1af-5aed-4b69-ae98-dc290370f6dc" containerID="c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864" exitCode=137 Jul 06 13:01:30.529147 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.529120 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" Jul 06 13:01:30.529329 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.529147 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" event={"ID":"ac6dd1af-5aed-4b69-ae98-dc290370f6dc","Type":"ContainerDied","Data":"c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864"} Jul 06 13:01:30.529329 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.529190 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr" event={"ID":"ac6dd1af-5aed-4b69-ae98-dc290370f6dc","Type":"ContainerDied","Data":"d5d8afc4e8c5ff8ae85251b1f3b9cc841ca71e8c5bc7c720c2922ea7540a54b4"} Jul 06 13:01:30.529329 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.529212 2572 scope.go:117] "RemoveContainer" containerID="c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864" Jul 06 13:01:30.544827 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.544806 2572 scope.go:117] "RemoveContainer" containerID="c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864" Jul 06 13:01:30.545130 ip-10-0-136-228 kubenswrapper[2572]: E0706 13:01:30.545108 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864\": container with ID starting with c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864 not found: ID does not exist" containerID="c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864" Jul 06 13:01:30.545208 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.545138 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864"} err="failed to get container status \"c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864\": rpc error: code = NotFound desc = could not find container \"c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864\": container with ID starting with c1c1d7a10b2ff0fc73a43356ed74ddda7f136bba7f9d395f5e455c3afbf4b864 not found: ID does not exist" Jul 06 13:01:30.557397 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.557369 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr"] Jul 06 13:01:30.561222 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:30.561202 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-984f35-29722380-vgkzr"] Jul 06 13:01:31.831607 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:31.831572 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac6dd1af-5aed-4b69-ae98-dc290370f6dc" path="/var/lib/kubelet/pods/ac6dd1af-5aed-4b69-ae98-dc290370f6dc/volumes" Jul 06 13:01:36.387185 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.387154 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm"] Jul 06 13:01:36.387711 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.387614 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6dd1af-5aed-4b69-ae98-dc290370f6dc" containerName="cleanup" Jul 06 13:01:36.387711 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.387634 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6dd1af-5aed-4b69-ae98-dc290370f6dc" containerName="cleanup" Jul 06 13:01:36.387855 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.387739 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6dd1af-5aed-4b69-ae98-dc290370f6dc" containerName="cleanup" Jul 06 13:01:36.391921 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.391900 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.396748 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.396723 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"kube-root-ca.crt\"" Jul 06 13:01:36.398151 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.398084 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"rate-test-model-984f35-kserve-self-signed-certs\"" Jul 06 13:01:36.399636 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.399127 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"openshift-service-ca.crt\"" Jul 06 13:01:36.402147 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.402127 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-984f35\"/\"default-dockercfg-phhqd\"" Jul 06 13:01:36.409265 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.409243 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm"] Jul 06 13:01:36.525201 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.525113 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-tmp-dir\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.525396 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.525278 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrbq\" (UniqueName: \"kubernetes.io/projected/64009898-7a28-4214-94fa-63bec7edbcd7-kube-api-access-tnrbq\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.525396 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.525325 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-model-cache\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.525396 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.525368 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-home\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.525547 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.525418 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-dshm\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.525547 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.525486 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64009898-7a28-4214-94fa-63bec7edbcd7-tls-certs\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.525797 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.525775 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:01:36.626107 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.626075 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64009898-7a28-4214-94fa-63bec7edbcd7-tls-certs\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.626289 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.626117 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-tmp-dir\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.626289 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.626161 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrbq\" (UniqueName: \"kubernetes.io/projected/64009898-7a28-4214-94fa-63bec7edbcd7-kube-api-access-tnrbq\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.626289 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.626185 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-model-cache\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.626289 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.626209 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-home\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.626551 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.626380 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-dshm\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.626627 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.626609 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-model-cache\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.626758 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.626733 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-home\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.626871 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.626753 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-tmp-dir\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.628443 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.628422 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-dshm\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.628567 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.628550 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64009898-7a28-4214-94fa-63bec7edbcd7-tls-certs\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.639687 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.639620 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrbq\" (UniqueName: \"kubernetes.io/projected/64009898-7a28-4214-94fa-63bec7edbcd7-kube-api-access-tnrbq\") pod \"rate-test-model-984f35-kserve-655cbf7dc5-78wsm\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.701504 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.701474 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:36.853784 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:36.853759 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm"] Jul 06 13:01:36.855834 ip-10-0-136-228 kubenswrapper[2572]: W0706 13:01:36.855805 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64009898_7a28_4214_94fa_63bec7edbcd7.slice/crio-1fdf2799c004742a76cce507de995ecd61e6d67edeed810de05e72586fe91ba0 WatchSource:0}: Error finding container 1fdf2799c004742a76cce507de995ecd61e6d67edeed810de05e72586fe91ba0: Status 404 returned error can't find the container with id 1fdf2799c004742a76cce507de995ecd61e6d67edeed810de05e72586fe91ba0 Jul 06 13:01:37.559928 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:37.559887 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" event={"ID":"64009898-7a28-4214-94fa-63bec7edbcd7","Type":"ContainerStarted","Data":"1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40"} Jul 06 13:01:37.559928 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:37.559931 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" event={"ID":"64009898-7a28-4214-94fa-63bec7edbcd7","Type":"ContainerStarted","Data":"1fdf2799c004742a76cce507de995ecd61e6d67edeed810de05e72586fe91ba0"} Jul 06 13:01:37.592397 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:37.592329 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" podStartSLOduration=1.5923091660000002 podStartE2EDuration="1.592309166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:01:37.590877318 +0000 UTC m=+2313.669425357" watchObservedRunningTime="2026-07-06 13:01:37.592309166 +0000 UTC m=+2313.670857193" Jul 06 13:01:40.421104 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:40.421065 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 13:01:41.020452 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:41.020415 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:01:41.131012 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:41.130962 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:01:46.702053 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:46.701988 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:46.702053 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:46.702063 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:46.714574 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:46.714544 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:47.615997 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:47.615968 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:01:50.630266 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:50.630229 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:01:50.730996 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:50.730963 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:01:54.661437 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:01:54.661399 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 13:02:09.147705 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:09.147670 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 13:02:17.034400 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:17.034361 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp"] Jul 06 13:02:17.089694 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:17.089640 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:17.949361 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:17.949325 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm"] Jul 06 13:02:17.977982 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:17.977942 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:19.076273 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:19.076236 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:19.467043 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:19.467002 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:19.576135 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:19.576093 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:19.674109 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:19.674071 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:19.890257 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:19.890204 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:23.701406 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:23.701375 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 13:02:24.552864 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:24.552831 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:32.053685 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.053622 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" podUID="9c51d897-9370-4f19-ae80-edf5d233748b" containerName="main" containerID="cri-o://7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a" gracePeriod=45 Jul 06 13:02:32.308254 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.308194 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:02:32.423604 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.423578 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgd7d\" (UniqueName: \"kubernetes.io/projected/9c51d897-9370-4f19-ae80-edf5d233748b-kube-api-access-rgd7d\") pod \"9c51d897-9370-4f19-ae80-edf5d233748b\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " Jul 06 13:02:32.423798 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.423680 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-home\") pod \"9c51d897-9370-4f19-ae80-edf5d233748b\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " Jul 06 13:02:32.423798 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.423774 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-dshm\") pod \"9c51d897-9370-4f19-ae80-edf5d233748b\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " Jul 06 13:02:32.423922 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.423808 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-tmp-dir\") pod \"9c51d897-9370-4f19-ae80-edf5d233748b\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " Jul 06 13:02:32.423922 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.423855 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c51d897-9370-4f19-ae80-edf5d233748b-tls-certs\") pod \"9c51d897-9370-4f19-ae80-edf5d233748b\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " Jul 06 13:02:32.423922 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.423879 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-home" (OuterVolumeSpecName: "home") pod "9c51d897-9370-4f19-ae80-edf5d233748b" (UID: "9c51d897-9370-4f19-ae80-edf5d233748b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:02:32.423922 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.423890 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-model-cache\") pod \"9c51d897-9370-4f19-ae80-edf5d233748b\" (UID: \"9c51d897-9370-4f19-ae80-edf5d233748b\") " Jul 06 13:02:32.424149 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.424081 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c51d897-9370-4f19-ae80-edf5d233748b" (UID: "9c51d897-9370-4f19-ae80-edf5d233748b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:02:32.424213 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.424177 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c51d897-9370-4f19-ae80-edf5d233748b" (UID: "9c51d897-9370-4f19-ae80-edf5d233748b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:02:32.424268 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.424183 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:32.424268 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.424249 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:32.425763 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.425736 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c51d897-9370-4f19-ae80-edf5d233748b-kube-api-access-rgd7d" (OuterVolumeSpecName: "kube-api-access-rgd7d") pod "9c51d897-9370-4f19-ae80-edf5d233748b" (UID: "9c51d897-9370-4f19-ae80-edf5d233748b"). InnerVolumeSpecName "kube-api-access-rgd7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:02:32.425878 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.425806 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-dshm" (OuterVolumeSpecName: "dshm") pod "9c51d897-9370-4f19-ae80-edf5d233748b" (UID: "9c51d897-9370-4f19-ae80-edf5d233748b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:02:32.425878 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.425845 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c51d897-9370-4f19-ae80-edf5d233748b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c51d897-9370-4f19-ae80-edf5d233748b" (UID: "9c51d897-9370-4f19-ae80-edf5d233748b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:02:32.525453 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.525422 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:32.525453 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.525450 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c51d897-9370-4f19-ae80-edf5d233748b-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:32.525453 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.525460 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c51d897-9370-4f19-ae80-edf5d233748b-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:32.525698 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.525469 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgd7d\" (UniqueName: \"kubernetes.io/projected/9c51d897-9370-4f19-ae80-edf5d233748b-kube-api-access-rgd7d\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:32.802333 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.802293 2572 generic.go:356] "Generic (PLEG): container finished" podID="9c51d897-9370-4f19-ae80-edf5d233748b" containerID="7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a" exitCode=0 Jul 06 13:02:32.802523 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.802337 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" Jul 06 13:02:32.802523 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.802383 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" event={"ID":"9c51d897-9370-4f19-ae80-edf5d233748b","Type":"ContainerDied","Data":"7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a"} Jul 06 13:02:32.802523 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.802428 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp" event={"ID":"9c51d897-9370-4f19-ae80-edf5d233748b","Type":"ContainerDied","Data":"c0319f615f8a6760e402f9d583c92887caef4d2eb3cb7db294daeb811c8869a0"} Jul 06 13:02:32.802523 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.802447 2572 scope.go:117] "RemoveContainer" containerID="7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a" Jul 06 13:02:32.810719 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.810700 2572 scope.go:117] "RemoveContainer" containerID="7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a" Jul 06 13:02:32.810970 ip-10-0-136-228 kubenswrapper[2572]: E0706 13:02:32.810953 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a\": container with ID starting with 7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a not found: ID does not exist" containerID="7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a" Jul 06 13:02:32.811020 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.810979 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a"} err="failed to get container status \"7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a\": rpc error: code = NotFound desc = could not find container \"7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a\": container with ID starting with 7e418f1a9ac2e80d571b5d1a89702c7d251d830295c5236ac9bf2be896f5623a not found: ID does not exist" Jul 06 13:02:32.824361 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.824339 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp"] Jul 06 13:02:32.828254 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.828234 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e47695/rate-test-model-e47695-kserve-86ddd8c79f-chwgp"] Jul 06 13:02:32.879054 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.879015 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:32.950739 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.950708 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:02:32.968564 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:32.968515 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" podUID="64009898-7a28-4214-94fa-63bec7edbcd7" containerName="main" containerID="cri-o://1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40" gracePeriod=45 Jul 06 13:02:33.209784 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.209762 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:02:33.332516 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.332427 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnrbq\" (UniqueName: \"kubernetes.io/projected/64009898-7a28-4214-94fa-63bec7edbcd7-kube-api-access-tnrbq\") pod \"64009898-7a28-4214-94fa-63bec7edbcd7\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " Jul 06 13:02:33.332516 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.332470 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-model-cache\") pod \"64009898-7a28-4214-94fa-63bec7edbcd7\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " Jul 06 13:02:33.332516 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.332517 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-tmp-dir\") pod \"64009898-7a28-4214-94fa-63bec7edbcd7\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " Jul 06 13:02:33.332791 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.332537 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-dshm\") pod \"64009898-7a28-4214-94fa-63bec7edbcd7\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " Jul 06 13:02:33.332791 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.332596 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64009898-7a28-4214-94fa-63bec7edbcd7-tls-certs\") pod \"64009898-7a28-4214-94fa-63bec7edbcd7\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " Jul 06 13:02:33.332791 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.332650 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-home\") pod \"64009898-7a28-4214-94fa-63bec7edbcd7\" (UID: \"64009898-7a28-4214-94fa-63bec7edbcd7\") " Jul 06 13:02:33.332959 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.332827 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-model-cache" (OuterVolumeSpecName: "model-cache") pod "64009898-7a28-4214-94fa-63bec7edbcd7" (UID: "64009898-7a28-4214-94fa-63bec7edbcd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:02:33.332959 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.332838 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64009898-7a28-4214-94fa-63bec7edbcd7" (UID: "64009898-7a28-4214-94fa-63bec7edbcd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:02:33.333067 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.332980 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:33.333067 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.333002 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:33.333067 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.333004 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-home" (OuterVolumeSpecName: "home") pod "64009898-7a28-4214-94fa-63bec7edbcd7" (UID: "64009898-7a28-4214-94fa-63bec7edbcd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:02:33.334704 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.334683 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-dshm" (OuterVolumeSpecName: "dshm") pod "64009898-7a28-4214-94fa-63bec7edbcd7" (UID: "64009898-7a28-4214-94fa-63bec7edbcd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:02:33.334821 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.334803 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64009898-7a28-4214-94fa-63bec7edbcd7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64009898-7a28-4214-94fa-63bec7edbcd7" (UID: "64009898-7a28-4214-94fa-63bec7edbcd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:02:33.334863 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.334829 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64009898-7a28-4214-94fa-63bec7edbcd7-kube-api-access-tnrbq" (OuterVolumeSpecName: "kube-api-access-tnrbq") pod "64009898-7a28-4214-94fa-63bec7edbcd7" (UID: "64009898-7a28-4214-94fa-63bec7edbcd7"). InnerVolumeSpecName "kube-api-access-tnrbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:02:33.433843 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.433810 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrbq\" (UniqueName: \"kubernetes.io/projected/64009898-7a28-4214-94fa-63bec7edbcd7-kube-api-access-tnrbq\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:33.433843 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.433836 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:33.433843 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.433845 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64009898-7a28-4214-94fa-63bec7edbcd7-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:33.433843 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.433856 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64009898-7a28-4214-94fa-63bec7edbcd7-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\"" Jul 06 13:02:33.807646 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.807604 2572 generic.go:356] "Generic (PLEG): container finished" podID="64009898-7a28-4214-94fa-63bec7edbcd7" containerID="1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40" exitCode=0 Jul 06 13:02:33.807852 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.807679 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" Jul 06 13:02:33.807852 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.807695 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" event={"ID":"64009898-7a28-4214-94fa-63bec7edbcd7","Type":"ContainerDied","Data":"1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40"} Jul 06 13:02:33.807852 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.807748 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm" event={"ID":"64009898-7a28-4214-94fa-63bec7edbcd7","Type":"ContainerDied","Data":"1fdf2799c004742a76cce507de995ecd61e6d67edeed810de05e72586fe91ba0"} Jul 06 13:02:33.807852 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.807766 2572 scope.go:117] "RemoveContainer" containerID="1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40" Jul 06 13:02:33.816314 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.816297 2572 scope.go:117] "RemoveContainer" containerID="1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40" Jul 06 13:02:33.816548 ip-10-0-136-228 kubenswrapper[2572]: E0706 13:02:33.816529 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40\": container with ID starting with 1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40 not found: ID does not exist" containerID="1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40" Jul 06 13:02:33.816614 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.816553 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40"} err="failed to get container status \"1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40\": rpc error: code = NotFound desc = could not find container \"1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40\": container with ID starting with 1803a096870e6681b42e14e1a76f4808168935ae7ad2f17ad4d667c434744b40 not found: ID does not exist" Jul 06 13:02:33.832125 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.832100 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c51d897-9370-4f19-ae80-edf5d233748b" path="/var/lib/kubelet/pods/9c51d897-9370-4f19-ae80-edf5d233748b/volumes" Jul 06 13:02:33.832370 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.832357 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm"] Jul 06 13:02:33.832797 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:33.832780 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-984f35/rate-test-model-984f35-kserve-655cbf7dc5-78wsm"] Jul 06 13:02:35.830450 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:35.830417 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64009898-7a28-4214-94fa-63bec7edbcd7" path="/var/lib/kubelet/pods/64009898-7a28-4214-94fa-63bec7edbcd7/volumes" Jul 06 13:02:39.566281 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:02:39.566245 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:02.746645 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:02.746607 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:03.018503 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:03.018416 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:03.485612 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:03.485556 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:03.929549 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:03.929445 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:04.029211 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:04.029177 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:04.129769 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:04.129732 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:04.343078 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:04.343037 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:06.837567 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:06.837530 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ksj9v"] Jul 06 13:03:10.445449 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:10.445409 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:10.545748 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:10.545719 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:10.717180 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:10.717140 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:11.419777 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:11.419745 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-774458dd6d-r66pd_fe4b7728-a734-44d7-93bd-6a97b8a70cc1/authorino/0.log" Jul 06 13:03:16.870063 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:16.870028 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67cf9968b4-dvbh9_b864c88a-0dce-4d8b-8a55-0259b4617bdd/manager/0.log" Jul 06 13:03:16.876545 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:16.876520 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67cf9968b4-dvbh9_b864c88a-0dce-4d8b-8a55-0259b4617bdd/copy-manifests/0.log" Jul 06 13:03:18.519936 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:18.519906 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-774458dd6d-r66pd_fe4b7728-a734-44d7-93bd-6a97b8a70cc1/authorino/0.log" Jul 06 13:03:18.884049 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:18.883976 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-sqv7w_53fbd44b-b0d2-490e-9c16-c3b0de832a6f/kuadrant-console-plugin/0.log" Jul 06 13:03:19.263085 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:19.263044 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ksj9v_d43bff14-4240-430d-972d-605f24dcf10b/limitador/0.log" Jul 06 13:03:20.126986 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:20.126902 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d7ff6d984-kwvk5_7ccc16fc-9866-4c75-8559-6717f17ddde3/kube-auth-proxy/0.log" Jul 06 13:03:21.469004 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:21.468973 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-nv678_58d8239f-b074-4c14-b9d1-ab3ac882a7c7/main/0.log" Jul 06 13:03:26.004034 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:26.004001 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:26.103859 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:26.103767 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:29.265135 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:29.265099 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8x6cj_40839c53-c59a-46b6-8299-4a2ed1c4ec90/global-pull-secret-syncer/0.log" Jul 06 13:03:29.382991 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:29.382963 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" Jul 06 13:03:29.522613 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:29.522522 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-228.ec2.internal_9bb0dda8697e373c519f6dc1fe8e110b/haproxy/0.log" Jul 06 13:03:35.114815 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:35.114784 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-774458dd6d-r66pd_fe4b7728-a734-44d7-93bd-6a97b8a70cc1/authorino/0.log" Jul 06 13:03:35.219891 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:35.219810 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-sqv7w_53fbd44b-b0d2-490e-9c16-c3b0de832a6f/kuadrant-console-plugin/0.log" Jul 06 13:03:35.474551 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:35.474468 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ksj9v_d43bff14-4240-430d-972d-605f24dcf10b/limitador/0.log" Jul 06 13:03:37.416876 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:37.416839 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-sgkbc_f0caf416-8c07-442a-8c58-f06069227b50/cluster-monitoring-operator/0.log" Jul 06 13:03:37.993439 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:37.993404 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jj89d_dade6ad6-8494-4c81-b45c-e14cc7b48e29/node-exporter/0.log" Jul 06 13:03:38.059142 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.059116 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jj89d_dade6ad6-8494-4c81-b45c-e14cc7b48e29/kube-rbac-proxy/0.log" Jul 06 13:03:38.093476 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.093451 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jj89d_dade6ad6-8494-4c81-b45c-e14cc7b48e29/init-textfile/0.log" Jul 06 13:03:38.249463 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.249367 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e9e2467-f132-46c3-b7ee-75ec491effd8/prometheus/0.log" Jul 06 13:03:38.304530 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.304500 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e9e2467-f132-46c3-b7ee-75ec491effd8/config-reloader/0.log" Jul 06 13:03:38.377226 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.377201 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e9e2467-f132-46c3-b7ee-75ec491effd8/thanos-sidecar/0.log" Jul 06 13:03:38.446056 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.446035 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e9e2467-f132-46c3-b7ee-75ec491effd8/kube-rbac-proxy-web/0.log" Jul 06 13:03:38.480677 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.480631 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e9e2467-f132-46c3-b7ee-75ec491effd8/kube-rbac-proxy/0.log" Jul 06 13:03:38.512581 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.512516 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e9e2467-f132-46c3-b7ee-75ec491effd8/kube-rbac-proxy-thanos/0.log" Jul 06 13:03:38.542368 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.542348 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e9e2467-f132-46c3-b7ee-75ec491effd8/init-config-reloader/0.log" Jul 06 13:03:38.669365 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.669338 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9c9fcffb4-rfpg8_523e67d1-8f4e-4fa2-8421-c083dc84dd64/telemeter-client/0.log" Jul 06 13:03:38.700714 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.700692 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9c9fcffb4-rfpg8_523e67d1-8f4e-4fa2-8421-c083dc84dd64/reload/0.log" Jul 06 13:03:38.731747 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:38.731728 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9c9fcffb4-rfpg8_523e67d1-8f4e-4fa2-8421-c083dc84dd64/kube-rbac-proxy/0.log" Jul 06 13:03:39.074401 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.074373 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j"] Jul 06 13:03:39.074727 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.074713 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64009898-7a28-4214-94fa-63bec7edbcd7" containerName="main" Jul 06 13:03:39.074791 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.074733 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="64009898-7a28-4214-94fa-63bec7edbcd7" containerName="main" Jul 06 13:03:39.074791 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.074757 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c51d897-9370-4f19-ae80-edf5d233748b" containerName="main" Jul 06 13:03:39.074791 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.074763 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c51d897-9370-4f19-ae80-edf5d233748b" containerName="main" Jul 06 13:03:39.074886 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.074817 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c51d897-9370-4f19-ae80-edf5d233748b" containerName="main" Jul 06 13:03:39.074886 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.074828 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="64009898-7a28-4214-94fa-63bec7edbcd7" containerName="main" Jul 06 13:03:39.077622 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.077606 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.091712 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.091688 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zfwcs\"/\"default-dockercfg-h759v\"" Jul 06 13:03:39.098962 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.098936 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j"] Jul 06 13:03:39.099866 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.099847 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfwcs\"/\"openshift-service-ca.crt\"" Jul 06 13:03:39.102180 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.102164 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfwcs\"/\"kube-root-ca.crt\"" Jul 06 13:03:39.172888 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.172859 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-sys\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.173022 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.172896 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx8p8\" (UniqueName: \"kubernetes.io/projected/ec898f8b-611e-4518-83d0-72ceec948d9e-kube-api-access-hx8p8\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.173022 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.172928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-proc\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.173022 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.173008 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-podres\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.173160 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.173090 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-lib-modules\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.274415 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.274376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8p8\" (UniqueName: \"kubernetes.io/projected/ec898f8b-611e-4518-83d0-72ceec948d9e-kube-api-access-hx8p8\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.274583 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.274437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-proc\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.274583 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.274469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-podres\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.274583 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.274528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-lib-modules\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.274773 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.274592 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-proc\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.274773 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.274591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-sys\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.274773 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.274641 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-sys\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.274773 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.274645 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-podres\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.274773 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.274686 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec898f8b-611e-4518-83d0-72ceec948d9e-lib-modules\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.283636 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.283613 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8p8\" (UniqueName: \"kubernetes.io/projected/ec898f8b-611e-4518-83d0-72ceec948d9e-kube-api-access-hx8p8\") pod \"perf-node-gather-daemonset-qbj2j\" (UID: \"ec898f8b-611e-4518-83d0-72ceec948d9e\") " pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.387350 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.387274 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:39.524137 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.524105 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j"] Jul 06 13:03:39.526983 ip-10-0-136-228 kubenswrapper[2572]: W0706 13:03:39.526948 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podec898f8b_611e_4518_83d0_72ceec948d9e.slice/crio-472f1d351896e9b46082a269781ae23d29146317236e4cbc1cc54a71090d7d3c WatchSource:0}: Error finding container 472f1d351896e9b46082a269781ae23d29146317236e4cbc1cc54a71090d7d3c: Status 404 returned error can't find the container with id 472f1d351896e9b46082a269781ae23d29146317236e4cbc1cc54a71090d7d3c Jul 06 13:03:39.775159 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:39.775120 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:40.081106 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:40.081003 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" event={"ID":"ec898f8b-611e-4518-83d0-72ceec948d9e","Type":"ContainerStarted","Data":"5070e2ceeb7ce040b19f13bb949b217783150cafe466561d3c4a14f6e03c60fb"} Jul 06 13:03:40.081280 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:40.081166 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:40.081280 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:40.081207 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" event={"ID":"ec898f8b-611e-4518-83d0-72ceec948d9e","Type":"ContainerStarted","Data":"472f1d351896e9b46082a269781ae23d29146317236e4cbc1cc54a71090d7d3c"} Jul 06 13:03:40.102039 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:40.101995 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" podStartSLOduration=1.101981262 podStartE2EDuration="1.101981262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:03:40.100274 +0000 UTC m=+2436.178822041" watchObservedRunningTime="2026-07-06 13:03:40.101981262 +0000 UTC m=+2436.180529289" Jul 06 13:03:41.186880 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:41.186838 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:43.076854 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:43.076826 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmvrc_a8b90c0d-5b63-442c-add0-54a38a3c435e/dns/0.log" Jul 06 13:03:43.103487 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:43.103456 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmvrc_a8b90c0d-5b63-442c-add0-54a38a3c435e/kube-rbac-proxy/0.log" Jul 06 13:03:43.184765 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:43.184737 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fbxl6_fddefa28-3b87-4803-833e-818c0d54b665/dns-node-resolver/0.log" Jul 06 13:03:43.850154 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:43.850128 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xqvf_5302ffb2-87cb-4aba-8e8a-0e7208e8b217/node-ca/0.log" Jul 06 13:03:45.245095 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:45.245059 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d7ff6d984-kwvk5_7ccc16fc-9866-4c75-8559-6717f17ddde3/kube-auth-proxy/0.log" Jul 06 13:03:45.531746 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:45.531606 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:45.644344 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:45.644293 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkr2l_21fedbcf-6e72-41a7-a196-aff0670ba1f3/konnectivity-agent/0.log" limit=1024 Jul 06 13:03:46.095711 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:46.095681 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zfwcs/perf-node-gather-daemonset-qbj2j" Jul 06 13:03:46.237830 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:46.237803 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vf2c7_ef5ec50a-7bdc-4ba5-9218-f15178c889b5/serve-healthcheck-canary/0.log" Jul 06 13:03:46.896959 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:46.896932 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lrf5l_c970c174-8210-443d-a38c-7542aeac0e1e/kube-rbac-proxy/0.log" Jul 06 13:03:46.924410 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:46.924388 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lrf5l_c970c174-8210-443d-a38c-7542aeac0e1e/exporter/0.log" Jul 06 13:03:46.960797 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:46.960773 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lrf5l_c970c174-8210-443d-a38c-7542aeac0e1e/extractor/0.log" Jul 06 13:03:49.628509 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:49.628456 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d7ff6d984-kwvk5_7ccc16fc-9866-4c75-8559-6717f17ddde3/kube-auth-proxy/0.log" Jul 06 13:03:50.171339 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:50.171309 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67cf9968b4-dvbh9_b864c88a-0dce-4d8b-8a55-0259b4617bdd/manager/0.log" Jul 06 13:03:50.193123 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:50.193092 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67cf9968b4-dvbh9_b864c88a-0dce-4d8b-8a55-0259b4617bdd/copy-manifests/0.log" Jul 06 13:03:51.347502 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:51.347474 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d7ff6d984-kwvk5_7ccc16fc-9866-4c75-8559-6717f17ddde3/kube-auth-proxy/0.log" Jul 06 13:03:51.639858 ip-10-0-136-228 kubenswrapper[2572]: I0706 13:03:51.639760 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69bb7bb478-knv5q_5634c8f1-5d2c-46ae-adf1-c0389b6950ce/manager/0.log"