Jul 06 23:56:01.931916 ip-10-0-133-6 systemd[1]: Starting Kubernetes Kubelet... Jul 06 23:56:02.340728 ip-10-0-133-6 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:56:02.340728 ip-10-0-133-6 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 23:56:02.340728 ip-10-0-133-6 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:56:02.340728 ip-10-0-133-6 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 23:56:02.340728 ip-10-0-133-6 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:56:02.341776 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.341653 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 23:56:02.349763 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349732 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:02.349763 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349760 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:02.349763 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349767 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349773 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349776 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349779 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349782 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349785 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349788 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349791 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349794 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349796 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349799 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349802 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349805 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349807 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349810 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349813 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349815 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349818 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349820 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:02.349930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349822 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349825 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349827 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349830 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349834 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349838 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349841 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349844 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349846 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349850 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349856 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349860 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349863 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349865 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349867 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349871 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349873 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349876 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349879 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349881 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:02.350385 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349884 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349886 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349889 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349891 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349894 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349896 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349899 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349901 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349904 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349906 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349909 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349911 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349913 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349916 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349919 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349921 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349924 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349926 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349929 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349932 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:02.350886 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349934 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349938 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349941 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349943 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349946 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349948 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349950 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349956 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349959 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349961 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349964 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349966 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349969 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349972 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349975 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349978 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349980 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349983 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349987 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:02.351368 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349991 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349995 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.349998 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350002 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350004 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350007 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350010 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350012 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350015 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350017 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350020 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350022 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350024 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350027 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350030 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350032 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350034 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350037 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350039 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350825 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350834 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:02.351841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350837 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350841 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350844 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350847 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350850 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350854 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350858 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350861 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350864 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350866 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350869 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350872 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350875 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350877 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350880 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350883 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350886 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350888 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350891 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:02.352349 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350893 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350896 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350899 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350902 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350904 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350907 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350911 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350915 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350918 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350921 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350924 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350928 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350931 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350934 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350937 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350940 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350943 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350946 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350948 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:02.352870 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350951 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350954 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350956 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350958 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350961 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350964 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350967 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350969 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350972 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350974 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350977 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350980 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350982 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350984 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350987 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350989 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350992 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350994 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.350999 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351001 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:02.353333 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351004 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351006 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351008 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351011 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351013 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351016 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351018 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351021 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351023 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351026 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351028 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351030 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351033 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351035 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351038 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351040 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351042 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351045 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351047 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351050 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:02.353839 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351053 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351055 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351057 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351060 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351063 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351065 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351068 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351070 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351073 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351075 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351079 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351082 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351084 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351086 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351089 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351091 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351094 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351096 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351099 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351509 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:02.354321 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351514 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351517 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351519 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351522 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351524 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351527 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351546 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351549 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351551 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351554 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351556 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351559 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351562 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351565 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351567 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351571 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351573 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351576 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351578 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351580 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:02.354841 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351583 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351585 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351590 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351592 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351595 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351597 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351600 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351602 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351605 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351609 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351611 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351614 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351616 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351619 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351621 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351624 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351626 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351629 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351631 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351633 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:02.355318 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351637 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351641 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351644 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351647 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351650 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351652 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351655 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351658 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351661 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351664 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351666 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351669 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351671 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351674 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351683 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351686 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351689 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351691 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351694 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351697 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:02.355905 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351700 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351703 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351706 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351709 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351712 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351714 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351717 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351720 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351722 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351725 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351727 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351730 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351733 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351737 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351740 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351742 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351745 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351747 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351749 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:02.356398 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351752 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351754 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351757 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351760 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351762 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351764 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351767 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351770 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351774 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351776 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351779 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351781 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351783 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351786 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351790 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351793 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351795 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351798 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.351800 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351874 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 23:56:02.356880 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351882 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351890 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351894 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351898 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351902 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351906 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351910 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351913 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351916 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351919 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351923 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351925 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351928 2563 flags.go:64] FLAG: --cgroup-root="" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351931 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351935 2563 flags.go:64] FLAG: --client-ca-file="" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351941 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351946 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351952 2563 flags.go:64] FLAG: --cluster-domain="" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351956 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351962 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351966 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351969 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351973 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 23:56:02.357377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351975 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351979 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351982 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351987 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351989 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351992 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351995 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.351998 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352002 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352005 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352008 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352011 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352013 2563 flags.go:64] FLAG: --enable-server="true" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352016 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352024 2563 flags.go:64] FLAG: --event-burst="100" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352027 2563 flags.go:64] FLAG: --event-qps="50" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352030 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352033 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352035 2563 flags.go:64] FLAG: --eviction-hard="" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352039 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352042 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352045 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352048 2563 flags.go:64] FLAG: --eviction-soft="" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352050 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352053 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 23:56:02.357979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352057 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352060 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352063 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352066 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352069 2563 flags.go:64] FLAG: --feature-gates="" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352072 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352075 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352078 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352081 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352084 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352089 2563 flags.go:64] FLAG: --help="false" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352092 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352095 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352097 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352100 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352103 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352107 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352109 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352112 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352114 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352117 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352120 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352123 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352125 2563 flags.go:64] FLAG: --kube-reserved="" Jul 06 23:56:02.358600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352128 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352131 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352134 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352136 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352139 2563 flags.go:64] FLAG: --lock-file="" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352142 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352146 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352148 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352154 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352157 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352160 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352162 2563 flags.go:64] FLAG: --logging-format="text" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352166 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352169 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352171 2563 flags.go:64] FLAG: --manifest-url="" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352174 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352180 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352183 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352189 2563 flags.go:64] FLAG: --max-pods="110" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352192 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352195 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352198 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352201 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352203 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352206 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 23:56:02.359239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352209 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352216 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352219 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352222 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352225 2563 flags.go:64] FLAG: --pod-cidr="" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352228 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352235 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352237 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352240 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352243 2563 flags.go:64] FLAG: --port="10250" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352246 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352249 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02a2a99da660bddae" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352252 2563 flags.go:64] FLAG: --qos-reserved="" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352255 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352257 2563 flags.go:64] FLAG: --register-node="true" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352260 2563 flags.go:64] FLAG: --register-with-taints="" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352264 2563 flags.go:64] FLAG: --registry-burst="10" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352267 2563 flags.go:64] FLAG: --registry-qps="5" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352270 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352273 2563 flags.go:64] FLAG: --reserved-memory="" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352277 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352280 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352282 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352285 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352288 2563 flags.go:64] FLAG: --runonce="false" Jul 06 23:56:02.359888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352290 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352294 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352297 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352300 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352303 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352305 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352308 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352311 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352314 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352317 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352319 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352331 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352334 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352337 2563 flags.go:64] FLAG: --system-cgroups="" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352340 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352346 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352349 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352352 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352356 2563 flags.go:64] FLAG: --tls-min-version="" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352362 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352364 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352367 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352370 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352373 2563 flags.go:64] FLAG: --v="2" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352377 2563 flags.go:64] FLAG: --version="false" Jul 06 23:56:02.360501 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352382 2563 flags.go:64] FLAG: --vmodule="" Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352386 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.352389 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352512 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352516 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352519 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352523 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352526 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352542 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352545 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352548 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352551 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352554 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352556 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352559 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352562 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352564 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352567 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352569 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352572 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:02.361109 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352574 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352577 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352580 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352582 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352585 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352587 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352591 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352593 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352596 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352599 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352601 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352604 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352606 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352615 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352618 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352620 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352623 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352625 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352628 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352630 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:02.361648 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352634 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352639 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352642 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352645 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352648 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352651 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352654 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352656 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352659 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352661 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352666 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352669 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352672 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352675 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352678 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352680 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352684 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352687 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:02.362131 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352691 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352694 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352696 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352699 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352702 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352705 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352707 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352710 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352718 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352721 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352723 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352726 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352728 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352730 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352735 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352737 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352739 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352742 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352744 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352747 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:02.362588 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352749 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352752 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352754 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352757 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352759 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352762 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352764 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352767 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352769 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352772 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352774 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352777 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352781 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352783 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352785 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352788 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352790 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352793 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352795 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352797 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:02.363062 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352800 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352803 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352806 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.352808 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.353623 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.360704 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.360721 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.360743 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.360749 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360807 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360812 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360816 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360818 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360821 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360823 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:02.363568 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360826 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360829 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360831 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360834 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360836 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360839 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360844 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360847 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360850 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360853 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360856 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360858 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360861 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360864 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360866 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360868 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360871 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360875 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360878 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:02.363943 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360881 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360884 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360886 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360889 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360893 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360895 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360898 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360900 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360903 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360905 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360908 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360910 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360913 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360916 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360919 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360921 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360924 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360926 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360929 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360931 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:02.364447 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360935 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360937 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360940 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360943 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360945 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360948 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360950 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360953 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360955 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360957 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360960 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360962 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360965 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360967 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360970 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360972 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360975 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360978 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360981 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360983 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:02.365108 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360986 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360988 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360991 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360993 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360996 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.360998 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361001 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361003 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361005 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361008 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361011 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361015 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361018 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361021 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361024 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361027 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361029 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361032 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361034 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361036 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:02.365624 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361039 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361041 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361043 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361046 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361048 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361051 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361054 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361057 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361060 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361062 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361065 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361068 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361070 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361073 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:02.366101 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.361078 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361196 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361201 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361204 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361207 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361210 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361212 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361216 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361220 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361223 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361226 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361229 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361231 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361234 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361236 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361239 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361242 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361244 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361247 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:56:02.366445 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361249 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361252 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361255 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361257 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361260 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361262 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361265 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361267 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361270 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361272 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361275 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361278 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361280 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361283 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361285 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361288 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361290 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361292 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361295 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361297 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361299 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:56:02.366931 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361302 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361304 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361307 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361309 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361311 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361314 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361316 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361319 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361321 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361324 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361326 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361328 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361331 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361334 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361336 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361338 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361341 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361343 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361346 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:56:02.367440 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361348 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361351 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361353 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361356 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361359 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361361 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361364 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361366 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361369 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361371 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361373 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361376 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361379 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361382 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361386 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361388 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361391 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361393 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361396 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361398 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:56:02.367938 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361401 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361403 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361405 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361408 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361410 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361413 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361415 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361418 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361420 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361422 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361425 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361427 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361430 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361432 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361435 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361437 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361440 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361443 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361445 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361448 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:56:02.368421 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:02.361450 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:56:02.368978 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.361455 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:56:02.368978 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.361593 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 23:56:02.368978 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.363786 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 23:56:02.368978 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.365330 2563 server.go:1019] "Starting client certificate rotation" Jul 06 23:56:02.368978 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.365429 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:56:02.368978 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.365477 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:56:02.368978 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.367950 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:56:02.369266 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.369247 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:56:02.378280 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.378259 2563 log.go:25] "Validated CRI v1 runtime API" Jul 06 23:56:02.380362 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.380347 2563 log.go:25] "Validated CRI v1 image API" Jul 06 23:56:02.383391 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.383376 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 23:56:02.389170 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.389136 2563 fs.go:135] Filesystem UUIDs: map[2ee61f84-1a4a-4717-a708-febc1b1e3d69:/dev/nvme0n1p4 3f7da5ad-f267-4b00-adc1-c80db60237b0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 06 23:56:02.389170 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.389160 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 23:56:02.395468 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.395329 2563 manager.go:216] Machine: {Timestamp:2026-07-06 23:56:02.393893368 +0000 UTC m=+0.339478731 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098405 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ae971e73ab62a55ec098a811a9551 SystemUUID:ec2ae971-e73a-b62a-55ec-098a811a9551 BootID:cf45e826-a4d9-4b94-943c-6caa9fcc53c0 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:21:19:c8:2c:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:21:19:c8:2c:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:77:28:ee:c8:19 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 23:56:02.395468 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.395451 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 23:56:02.395680 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.395594 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 23:56:02.399366 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.399325 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 23:56:02.399599 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.399372 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 23:56:02.399673 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.399582 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:56:02.400818 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.400804 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 23:56:02.400849 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.400820 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 23:56:02.400849 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.400829 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:56:02.401627 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.401612 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:56:02.403489 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.403476 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 23:56:02.406593 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.406578 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:56:02.406735 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.406725 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 23:56:02.406772 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.406758 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 06 23:56:02.406772 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.406769 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 23:56:02.406837 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.406783 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 23:56:02.406837 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.406793 2563 kubelet.go:398] "Adding apiserver pod source" Jul 06 23:56:02.406837 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.406804 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 23:56:02.407205 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.407189 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:56:02.407265 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.407224 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:56:02.410038 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.410011 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 23:56:02.412422 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.412036 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 23:56:02.412499 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.412438 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 23:56:02.413602 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413591 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 23:56:02.413641 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413606 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 23:56:02.413641 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413612 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 23:56:02.413641 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413617 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 23:56:02.413641 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413627 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 23:56:02.413641 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413634 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 23:56:02.413771 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413643 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 23:56:02.413771 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413649 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 23:56:02.413771 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413655 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 23:56:02.413771 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413660 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 23:56:02.413771 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413675 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 23:56:02.413771 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.413680 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 23:56:02.415555 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.415544 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 23:56:02.415597 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.415559 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 23:56:02.419028 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.418993 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-6.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 23:56:02.419137 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.419118 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 23:56:02.419185 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.419119 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 23:56:02.419185 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.419157 2563 server.go:1268] "Started kubelet" Jul 06 23:56:02.419480 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.419398 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 23:56:02.420229 ip-10-0-133-6 systemd[1]: Started Kubernetes Kubelet. Jul 06 23:56:02.422046 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.419437 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 23:56:02.422238 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.422222 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 23:56:02.424566 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.424515 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 06 23:56:02.424764 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.424738 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 23:56:02.424896 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.424849 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 23:56:02.428749 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.427881 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-6.ec2.internal.18bfd8a0c1c287dd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-6.ec2.internal,UID:ip-10-0-133-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-6.ec2.internal,},FirstTimestamp:2026-07-06 23:56:02.419124189 +0000 UTC m=+0.364709602,LastTimestamp:2026-07-06 23:56:02.419124189 +0000 UTC m=+0.364709602,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-6.ec2.internal,}" Jul 06 23:56:02.429492 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.429471 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 23:56:02.429685 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.429473 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 23:56:02.429685 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.429608 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:02.430184 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.430169 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 23:56:02.430264 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.430255 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 23:56:02.430373 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.430297 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 23:56:02.430470 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.430456 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 06 23:56:02.430522 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.430471 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 06 23:56:02.430788 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.430768 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 23:56:02.430788 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.430790 2563 factory.go:55] Registering systemd factory Jul 06 23:56:02.431083 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.430798 2563 factory.go:223] Registration of the systemd container factory successfully Jul 06 23:56:02.431083 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.431026 2563 factory.go:153] Registering CRI-O factory Jul 06 23:56:02.431083 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.431038 2563 factory.go:223] Registration of the crio container factory successfully Jul 06 23:56:02.431083 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.431067 2563 factory.go:103] Registering Raw factory Jul 06 23:56:02.431083 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.431087 2563 manager.go:1196] Started watching for new ooms in manager Jul 06 23:56:02.431457 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.431440 2563 manager.go:349] Starting recovery of all containers Jul 06 23:56:02.434786 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.434755 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 23:56:02.434918 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.434881 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 23:56:02.437382 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.437265 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-drvkd" Jul 06 23:56:02.442396 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.442334 2563 manager.go:354] Recovery completed Jul 06 23:56:02.445352 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.445331 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-drvkd" Jul 06 23:56:02.446463 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.446451 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:02.448522 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.448506 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:02.448601 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.448553 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:02.448601 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.448564 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:02.449121 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.449106 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 23:56:02.449177 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.449121 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:02.449212 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.449123 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 23:56:02.449212 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.449203 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:56:02.449873 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.449859 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:02.449935 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.449883 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:02.449935 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.449893 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:02.450169 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.450158 2563 policy_none.go:49] "None policy: Start" Jul 06 23:56:02.450198 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.450179 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 23:56:02.450198 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.450190 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 23:56:02.450913 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.450783 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-6.ec2.internal.18bfd8a0c3831c04 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-6.ec2.internal,UID:ip-10-0-133-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-6.ec2.internal,},FirstTimestamp:2026-07-06 23:56:02.448522244 +0000 UTC m=+0.394107607,LastTimestamp:2026-07-06 23:56:02.448522244 +0000 UTC m=+0.394107607,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-6.ec2.internal,}" Jul 06 23:56:02.451131 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.451117 2563 policy_none.go:47] "Start" Jul 06 23:56:02.483448 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.483208 2563 manager.go:337] "Starting Device Plugin manager" Jul 06 23:56:02.483625 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.483479 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 23:56:02.483625 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.483493 2563 server.go:87] "Starting device plugin registration server" Jul 06 23:56:02.483783 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.483771 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 23:56:02.483842 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.483786 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 23:56:02.483934 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.483919 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 23:56:02.484030 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.484018 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 23:56:02.484104 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.484031 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 23:56:02.484558 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.484522 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 23:56:02.484623 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.484587 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:02.568569 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.568511 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 23:56:02.570052 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.570037 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 23:56:02.570139 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.570074 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 23:56:02.570139 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.570097 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 23:56:02.570139 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.570129 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 23:56:02.572449 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.572430 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:02.584764 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.584743 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:02.585612 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.585595 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:02.585683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.585629 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:02.585683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.585641 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:02.585683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.585671 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.594361 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.594260 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.594361 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.594294 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-6.ec2.internal\": node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:02.607889 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.607865 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:02.670604 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.670566 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal"] Jul 06 23:56:02.670761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.670667 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:02.671674 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.671659 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:02.671736 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.671689 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:02.671736 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.671723 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:02.673014 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.673003 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:02.673161 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.673147 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.673197 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.673177 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:02.673751 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.673727 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:02.673751 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.673743 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:02.673846 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.673758 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:02.673846 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.673768 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:02.673846 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.673772 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:02.673846 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.673780 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:02.674843 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.674829 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.674939 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.674851 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:56:02.675465 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.675450 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:56:02.675566 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.675471 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:56:02.675566 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.675484 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:56:02.702381 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.702352 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-6.ec2.internal\" not found" node="ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.706970 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.706949 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-6.ec2.internal\" not found" node="ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.707941 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.707925 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:02.732343 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.732310 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef76d5125552ec29a9aa27d158114806-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal\" (UID: \"ef76d5125552ec29a9aa27d158114806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.732343 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.732348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d45b6b823d2fa18cb79d62947f7a1859-config\") pod \"kube-apiserver-proxy-ip-10-0-133-6.ec2.internal\" (UID: \"d45b6b823d2fa18cb79d62947f7a1859\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.732582 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.732364 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef76d5125552ec29a9aa27d158114806-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal\" (UID: \"ef76d5125552ec29a9aa27d158114806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.808667 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.808640 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:02.832989 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.832964 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef76d5125552ec29a9aa27d158114806-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal\" (UID: \"ef76d5125552ec29a9aa27d158114806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.833085 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.833002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef76d5125552ec29a9aa27d158114806-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal\" (UID: \"ef76d5125552ec29a9aa27d158114806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.833085 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.833019 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d45b6b823d2fa18cb79d62947f7a1859-config\") pod \"kube-apiserver-proxy-ip-10-0-133-6.ec2.internal\" (UID: \"d45b6b823d2fa18cb79d62947f7a1859\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.833085 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.833068 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d45b6b823d2fa18cb79d62947f7a1859-config\") pod \"kube-apiserver-proxy-ip-10-0-133-6.ec2.internal\" (UID: \"d45b6b823d2fa18cb79d62947f7a1859\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.833085 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.833072 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef76d5125552ec29a9aa27d158114806-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal\" (UID: \"ef76d5125552ec29a9aa27d158114806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.833208 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:02.833106 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef76d5125552ec29a9aa27d158114806-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal\" (UID: \"ef76d5125552ec29a9aa27d158114806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" Jul 06 23:56:02.909452 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:02.909370 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.004883 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.004852 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" Jul 06 23:56:03.009889 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:03.009871 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.009938 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.009907 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal" Jul 06 23:56:03.112354 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:03.112311 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.213198 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:03.213116 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.313767 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:03.313728 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.364228 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.364186 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 23:56:03.364759 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.364359 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:56:03.414236 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:03.414201 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.450233 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.450200 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 23:51:03 +0000 UTC" deadline="2027-12-20 04:46:07.017604367 +0000 UTC" Jul 06 23:56:03.450233 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.450233 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12748h50m3.567374888s" Jul 06 23:56:03.514936 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:03.514859 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.551463 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.551439 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:03.552416 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:03.552382 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45b6b823d2fa18cb79d62947f7a1859.slice/crio-6a38fbc8613e4799c52aad2e2982e928e7e1ae2054e403fd5273e11696005a66 WatchSource:0}: Error finding container 6a38fbc8613e4799c52aad2e2982e928e7e1ae2054e403fd5273e11696005a66: Status 404 returned error can't find the container with id 6a38fbc8613e4799c52aad2e2982e928e7e1ae2054e403fd5273e11696005a66 Jul 06 23:56:03.552710 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:03.552677 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef76d5125552ec29a9aa27d158114806.slice/crio-938364af458ae16ef17df128c913c5efe36ee26528a9ca54308b7dc48b69e0fe WatchSource:0}: Error finding container 938364af458ae16ef17df128c913c5efe36ee26528a9ca54308b7dc48b69e0fe: Status 404 returned error can't find the container with id 938364af458ae16ef17df128c913c5efe36ee26528a9ca54308b7dc48b69e0fe Jul 06 23:56:03.556396 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.556379 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:56:03.557959 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.557944 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:03.574231 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.574189 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal" event={"ID":"d45b6b823d2fa18cb79d62947f7a1859","Type":"ContainerStarted","Data":"6a38fbc8613e4799c52aad2e2982e928e7e1ae2054e403fd5273e11696005a66"} Jul 06 23:56:03.575319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.575295 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" event={"ID":"ef76d5125552ec29a9aa27d158114806","Type":"ContainerStarted","Data":"938364af458ae16ef17df128c913c5efe36ee26528a9ca54308b7dc48b69e0fe"} Jul 06 23:56:03.617813 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:03.617777 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.724442 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:03.724401 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.831651 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:03.831529 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-6.ec2.internal\" not found" Jul 06 23:56:03.904143 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.904107 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:03.930339 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.930294 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" Jul 06 23:56:03.942628 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.942602 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:56:03.944520 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.944500 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal" Jul 06 23:56:03.951417 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:03.951393 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:56:04.408102 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.408067 2563 apiserver.go:52] "Watching apiserver" Jul 06 23:56:04.422085 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.422045 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 23:56:04.424761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.424220 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-4qtnq","kube-system/konnectivity-agent-4rfjl","kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k","openshift-image-registry/node-ca-8jrnh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal","openshift-multus/multus-additional-cni-plugins-82nrx","openshift-network-operator/iptables-alerter-zrxrv","openshift-cluster-node-tuning-operator/tuned-v49gd","openshift-dns/node-resolver-5tmhb","openshift-multus/multus-jmpjk","openshift-multus/network-metrics-daemon-vsq4t","openshift-network-diagnostics/network-check-target-mkjcg","openshift-ovn-kubernetes/ovnkube-node-cwc59"] Jul 06 23:56:04.426324 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.426297 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.427291 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.427266 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:04.427416 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.427379 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:04.428646 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.428624 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.429258 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.429234 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 23:56:04.429437 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.429236 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 23:56:04.429437 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.429319 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 23:56:04.429437 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.429353 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 23:56:04.429437 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.429411 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mtjlm\"" Jul 06 23:56:04.429714 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.429594 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 23:56:04.430130 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.430096 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 23:56:04.434348 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.434322 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 23:56:04.434568 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.434544 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 23:56:04.435423 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.435294 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 23:56:04.435423 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.435351 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.437157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.435778 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.437157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.435806 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wjt67\"" Jul 06 23:56:04.437157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.435996 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 23:56:04.437157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.436063 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 23:56:04.437157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.436088 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 23:56:04.437157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.436287 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.437157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.436524 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.437157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.436584 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:04.437157 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.436714 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:04.437584 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.437375 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:04.440445 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.438517 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 23:56:04.440445 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.438942 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l2m9d\"" Jul 06 23:56:04.440445 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.438947 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 23:56:04.440445 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.439181 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 23:56:04.440445 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.439287 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.440777 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.440556 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441017 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441029 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441029 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441296 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441319 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441341 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441360 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441474 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441523 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441547 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z8kw2\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441593 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441626 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9h5rp\"" Jul 06 23:56:04.441761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.441476 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6b7w4\"" Jul 06 23:56:04.442440 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.442257 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:04.442440 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.442291 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8s2lb\"" Jul 06 23:56:04.442440 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.442320 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:04.442571 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.442468 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 23:56:04.442571 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.442491 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 23:56:04.442698 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.442671 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8g88m\"" Jul 06 23:56:04.442835 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.442814 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 23:56:04.442986 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.442970 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w6v94\"" Jul 06 23:56:04.443821 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.443796 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-sys\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.443921 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.443835 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-var-lib-kubelet\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.443921 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.443867 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-run-netns\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.443921 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.443891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-node-log\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.444082 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.443947 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-system-cni-dir\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.444082 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.443972 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-socket-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.444082 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.443995 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-etc-selinux\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.444082 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444017 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-lib-modules\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.444082 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444043 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7nvv\" (UniqueName: \"kubernetes.io/projected/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-kube-api-access-h7nvv\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.444082 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444066 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f1d07f90-5abf-453f-bc2a-920be5e548a8-dbus\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:04.444367 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444098 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbth\" (UniqueName: \"kubernetes.io/projected/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-kube-api-access-zgbth\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.444367 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444121 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa00bed5-0833-4872-99cb-7c9d933d9323-host\") pod \"node-ca-8jrnh\" (UID: \"aa00bed5-0833-4872-99cb-7c9d933d9323\") " pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.444367 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444145 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-run-openvswitch\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.444367 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/afda7b21-8a0a-440c-bf34-3dd094197ca7-ovn-node-metrics-cert\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.444367 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444201 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99c62bce-e3c3-4979-ac9e-5c3d187870df-agent-certs\") pod \"konnectivity-agent-4rfjl\" (UID: \"99c62bce-e3c3-4979-ac9e-5c3d187870df\") " pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:04.444367 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444230 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-device-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.444367 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444267 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-systemd\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.444367 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444301 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99c62bce-e3c3-4979-ac9e-5c3d187870df-konnectivity-ca\") pod \"konnectivity-agent-4rfjl\" (UID: \"99c62bce-e3c3-4979-ac9e-5c3d187870df\") " pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:04.444367 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-modprobe-d\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444366 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444430 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444466 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6kj\" (UniqueName: \"kubernetes.io/projected/aa00bed5-0833-4872-99cb-7c9d933d9323-kube-api-access-nb6kj\") pod \"node-ca-8jrnh\" (UID: \"aa00bed5-0833-4872-99cb-7c9d933d9323\") " pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444493 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-run\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444524 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/afda7b21-8a0a-440c-bf34-3dd094197ca7-ovnkube-config\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444549 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-sysctl-d\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f1d07f90-5abf-453f-bc2a-920be5e548a8-kubelet-config\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444696 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-os-release\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444726 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-slash\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444768 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-etc-openvswitch\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.444803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444802 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-log-socket\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444841 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-tuned\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444876 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-systemd-units\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-var-lib-openvswitch\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-run-ovn\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.444973 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwx4w\" (UniqueName: \"kubernetes.io/projected/afda7b21-8a0a-440c-bf34-3dd094197ca7-kube-api-access-zwx4w\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445005 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa00bed5-0833-4872-99cb-7c9d933d9323-serviceca\") pod \"node-ca-8jrnh\" (UID: \"aa00bed5-0833-4872-99cb-7c9d933d9323\") " pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445072 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43576a49-39c7-4f5e-9537-792cc49db45c-iptables-alerter-script\") pod \"iptables-alerter-zrxrv\" (UID: \"43576a49-39c7-4f5e-9537-792cc49db45c\") " pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445099 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-host\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445126 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-cni-bin\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445196 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brw7x\" (UniqueName: \"kubernetes.io/projected/ff834cf2-cbcd-4722-9c4e-597f34f52036-kube-api-access-brw7x\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-kubernetes\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-run-systemd\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445276 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/afda7b21-8a0a-440c-bf34-3dd094197ca7-ovnkube-script-lib\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445303 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff834cf2-cbcd-4722-9c4e-597f34f52036-cni-binary-copy\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.445702 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445326 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-cni-netd\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445390 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/afda7b21-8a0a-440c-bf34-3dd094197ca7-env-overrides\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445419 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-registration-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-sys-fs\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445498 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43576a49-39c7-4f5e-9537-792cc49db45c-host-slash\") pod \"iptables-alerter-zrxrv\" (UID: \"43576a49-39c7-4f5e-9537-792cc49db45c\") " pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445589 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-tmp\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445640 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-kubelet\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445680 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff834cf2-cbcd-4722-9c4e-597f34f52036-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-cnibin\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445740 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff834cf2-cbcd-4722-9c4e-597f34f52036-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445758 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzdzq\" (UniqueName: \"kubernetes.io/projected/43576a49-39c7-4f5e-9537-792cc49db45c-kube-api-access-qzdzq\") pod \"iptables-alerter-zrxrv\" (UID: \"43576a49-39c7-4f5e-9537-792cc49db45c\") " pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.445774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-sysctl-conf\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.446383 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.446297 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:04.449757 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.449209 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:56:04.449897 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.449797 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-sysconfig\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.468904 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.468871 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mq2qg" Jul 06 23:56:04.477982 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.477406 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mq2qg" Jul 06 23:56:04.531566 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.531533 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550274 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa00bed5-0833-4872-99cb-7c9d933d9323-serviceca\") pod \"node-ca-8jrnh\" (UID: \"aa00bed5-0833-4872-99cb-7c9d933d9323\") " pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43576a49-39c7-4f5e-9537-792cc49db45c-iptables-alerter-script\") pod \"iptables-alerter-zrxrv\" (UID: \"43576a49-39c7-4f5e-9537-792cc49db45c\") " pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550337 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-host\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550354 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550379 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-cni-bin\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550409 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2x6\" (UniqueName: \"kubernetes.io/projected/c4cb04e9-3bdd-4c7b-8b9f-8063c3605454-kube-api-access-dr2x6\") pod \"node-resolver-5tmhb\" (UID: \"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454\") " pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550438 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brw7x\" (UniqueName: \"kubernetes.io/projected/ff834cf2-cbcd-4722-9c4e-597f34f52036-kube-api-access-brw7x\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550463 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-kubernetes\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550492 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-run-systemd\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550518 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/afda7b21-8a0a-440c-bf34-3dd094197ca7-ovnkube-script-lib\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550547 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-run-k8s-cni-cncf-io\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-run-netns\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550623 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff834cf2-cbcd-4722-9c4e-597f34f52036-cni-binary-copy\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550649 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550674 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550712 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-cni-netd\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.554471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/afda7b21-8a0a-440c-bf34-3dd094197ca7-env-overrides\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550785 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-registration-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550820 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-sys-fs\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550847 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43576a49-39c7-4f5e-9537-792cc49db45c-host-slash\") pod \"iptables-alerter-zrxrv\" (UID: \"43576a49-39c7-4f5e-9537-792cc49db45c\") " pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-tmp\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550883 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-kubelet\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-conf-dir\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff834cf2-cbcd-4722-9c4e-597f34f52036-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550939 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-cnibin\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.550973 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff834cf2-cbcd-4722-9c4e-597f34f52036-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551009 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzdzq\" (UniqueName: \"kubernetes.io/projected/43576a49-39c7-4f5e-9537-792cc49db45c-kube-api-access-qzdzq\") pod \"iptables-alerter-zrxrv\" (UID: \"43576a49-39c7-4f5e-9537-792cc49db45c\") " pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551034 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-sysctl-conf\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-cnibin\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551073 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-etc-kubernetes\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-sysconfig\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551141 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-sys\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.555574 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551166 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-var-lib-kubelet\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551196 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4cb04e9-3bdd-4c7b-8b9f-8063c3605454-hosts-file\") pod \"node-resolver-5tmhb\" (UID: \"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454\") " pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551227 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-run-netns\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551256 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-node-log\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-system-cni-dir\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-socket-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-etc-selinux\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551385 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-lib-modules\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551417 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nvv\" (UniqueName: \"kubernetes.io/projected/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-kube-api-access-h7nvv\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551443 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f1d07f90-5abf-453f-bc2a-920be5e548a8-dbus\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-cni-dir\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551498 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-os-release\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551524 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02c4a893-1ad4-4389-81b7-9bb7ded935b6-cni-binary-copy\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551549 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-socket-dir-parent\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551576 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-var-lib-cni-multus\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551602 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-hostroot\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551649 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbth\" (UniqueName: \"kubernetes.io/projected/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-kube-api-access-zgbth\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.556412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551678 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa00bed5-0833-4872-99cb-7c9d933d9323-host\") pod \"node-ca-8jrnh\" (UID: \"aa00bed5-0833-4872-99cb-7c9d933d9323\") " pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551704 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-run-openvswitch\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551749 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/afda7b21-8a0a-440c-bf34-3dd094197ca7-ovn-node-metrics-cert\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551776 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4cb04e9-3bdd-4c7b-8b9f-8063c3605454-tmp-dir\") pod \"node-resolver-5tmhb\" (UID: \"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454\") " pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551804 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-var-lib-cni-bin\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551831 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99c62bce-e3c3-4979-ac9e-5c3d187870df-agent-certs\") pod \"konnectivity-agent-4rfjl\" (UID: \"99c62bce-e3c3-4979-ac9e-5c3d187870df\") " pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-device-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-systemd\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551911 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99c62bce-e3c3-4979-ac9e-5c3d187870df-konnectivity-ca\") pod \"konnectivity-agent-4rfjl\" (UID: \"99c62bce-e3c3-4979-ac9e-5c3d187870df\") " pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551930 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-modprobe-d\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551967 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.551995 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6kj\" (UniqueName: \"kubernetes.io/projected/aa00bed5-0833-4872-99cb-7c9d933d9323-kube-api-access-nb6kj\") pod \"node-ca-8jrnh\" (UID: \"aa00bed5-0833-4872-99cb-7c9d933d9323\") " pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552015 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-run\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/afda7b21-8a0a-440c-bf34-3dd094197ca7-ovnkube-config\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552049 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552092 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxr4\" (UniqueName: \"kubernetes.io/projected/305b651b-c1bc-4b0b-99fb-091bebcebb06-kube-api-access-cwxr4\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:04.557275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552118 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-sysctl-d\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552151 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f1d07f90-5abf-453f-bc2a-920be5e548a8-kubelet-config\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552177 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-os-release\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-slash\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552233 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-var-lib-kubelet\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552255 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-daemon-config\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-etc-openvswitch\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552305 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-log-socket\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552332 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-tuned\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552361 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-systemd-units\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552378 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-var-lib-openvswitch\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552395 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-run-ovn\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552415 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwx4w\" (UniqueName: \"kubernetes.io/projected/afda7b21-8a0a-440c-bf34-3dd094197ca7-kube-api-access-zwx4w\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552442 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-system-cni-dir\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552464 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-run-multus-certs\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.552487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94rw\" (UniqueName: \"kubernetes.io/projected/02c4a893-1ad4-4389-81b7-9bb7ded935b6-kube-api-access-q94rw\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553061 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-kubelet\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.558179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553108 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa00bed5-0833-4872-99cb-7c9d933d9323-serviceca\") pod \"node-ca-8jrnh\" (UID: \"aa00bed5-0833-4872-99cb-7c9d933d9323\") " pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553130 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-host\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553256 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553433 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-run\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553465 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43576a49-39c7-4f5e-9537-792cc49db45c-iptables-alerter-script\") pod \"iptables-alerter-zrxrv\" (UID: \"43576a49-39c7-4f5e-9537-792cc49db45c\") " pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553483 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-cni-bin\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553571 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-run-systemd\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553643 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-etc-openvswitch\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553694 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-log-socket\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554022 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554229 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff834cf2-cbcd-4722-9c4e-597f34f52036-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554239 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99c62bce-e3c3-4979-ac9e-5c3d187870df-konnectivity-ca\") pod \"konnectivity-agent-4rfjl\" (UID: \"99c62bce-e3c3-4979-ac9e-5c3d187870df\") " pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554302 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-kubernetes\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554302 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff834cf2-cbcd-4722-9c4e-597f34f52036-cni-binary-copy\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554341 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-modprobe-d\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554356 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-cnibin\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554394 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554437 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.559319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.553433 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f1d07f90-5abf-453f-bc2a-920be5e548a8-dbus\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554661 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-systemd\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554714 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-var-lib-kubelet\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554771 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43576a49-39c7-4f5e-9537-792cc49db45c-host-slash\") pod \"iptables-alerter-zrxrv\" (UID: \"43576a49-39c7-4f5e-9537-792cc49db45c\") " pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554822 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-registration-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554866 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-var-lib-openvswitch\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554832 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-sysctl-conf\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554890 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff834cf2-cbcd-4722-9c4e-597f34f52036-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554856 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-systemd-units\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554964 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-run-ovn\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554967 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f1d07f90-5abf-453f-bc2a-920be5e548a8-kubelet-config\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.555011 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-cni-netd\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.555054 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-sys\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.555034 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-os-release\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.555090 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-run-openvswitch\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.555676 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/afda7b21-8a0a-440c-bf34-3dd094197ca7-ovnkube-script-lib\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.555891 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:04.560409 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.555989 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret podName:f1d07f90-5abf-453f-bc2a-920be5e548a8 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:05.05594611 +0000 UTC m=+3.001531482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret") pod "global-pull-secret-syncer-4qtnq" (UID: "f1d07f90-5abf-453f-bc2a-920be5e548a8") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556000 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/afda7b21-8a0a-440c-bf34-3dd094197ca7-env-overrides\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.555055 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556119 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-sysconfig\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556135 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa00bed5-0833-4872-99cb-7c9d933d9323-host\") pod \"node-ca-8jrnh\" (UID: \"aa00bed5-0833-4872-99cb-7c9d933d9323\") " pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556170 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-slash\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556189 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff834cf2-cbcd-4722-9c4e-597f34f52036-system-cni-dir\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556234 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-host-run-netns\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556235 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-etc-selinux\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.554922 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-sys-fs\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556292 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-device-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556333 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/afda7b21-8a0a-440c-bf34-3dd094197ca7-node-log\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556383 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-lib-modules\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556401 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-socket-dir\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.556579 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-sysctl-d\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.557012 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/afda7b21-8a0a-440c-bf34-3dd094197ca7-ovnkube-config\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.559476 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-etc-tuned\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.559910 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/afda7b21-8a0a-440c-bf34-3dd094197ca7-ovn-node-metrics-cert\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.561550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.560894 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99c62bce-e3c3-4979-ac9e-5c3d187870df-agent-certs\") pod \"konnectivity-agent-4rfjl\" (UID: \"99c62bce-e3c3-4979-ac9e-5c3d187870df\") " pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:04.562534 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.561018 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:04.562534 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.561036 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:04.562534 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.561057 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xm59l for pod openshift-network-diagnostics/network-check-target-mkjcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:04.562534 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.561069 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-tmp\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.562534 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.561192 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l podName:612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc nodeName:}" failed. No retries permitted until 2026-07-06 23:56:05.061173494 +0000 UTC m=+3.006758868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xm59l" (UniqueName: "kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l") pod "network-check-target-mkjcg" (UID: "612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:04.562907 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.562883 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6kj\" (UniqueName: \"kubernetes.io/projected/aa00bed5-0833-4872-99cb-7c9d933d9323-kube-api-access-nb6kj\") pod \"node-ca-8jrnh\" (UID: \"aa00bed5-0833-4872-99cb-7c9d933d9323\") " pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.564009 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.563985 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brw7x\" (UniqueName: \"kubernetes.io/projected/ff834cf2-cbcd-4722-9c4e-597f34f52036-kube-api-access-brw7x\") pod \"multus-additional-cni-plugins-82nrx\" (UID: \"ff834cf2-cbcd-4722-9c4e-597f34f52036\") " pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.566108 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.566081 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7nvv\" (UniqueName: \"kubernetes.io/projected/c4d90ac1-1f6b-451c-85cd-5d9c30acfcec-kube-api-access-h7nvv\") pod \"tuned-v49gd\" (UID: \"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec\") " pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.570566 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.570505 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzdzq\" (UniqueName: \"kubernetes.io/projected/43576a49-39c7-4f5e-9537-792cc49db45c-kube-api-access-qzdzq\") pod \"iptables-alerter-zrxrv\" (UID: \"43576a49-39c7-4f5e-9537-792cc49db45c\") " pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.570936 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.570914 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbth\" (UniqueName: \"kubernetes.io/projected/f955c1d2-b48b-4ac9-aa50-bdff45fd5813-kube-api-access-zgbth\") pod \"aws-ebs-csi-driver-node-lm97k\" (UID: \"f955c1d2-b48b-4ac9-aa50-bdff45fd5813\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.571012 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.570993 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwx4w\" (UniqueName: \"kubernetes.io/projected/afda7b21-8a0a-440c-bf34-3dd094197ca7-kube-api-access-zwx4w\") pod \"ovnkube-node-cwc59\" (UID: \"afda7b21-8a0a-440c-bf34-3dd094197ca7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.653597 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.653534 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:04.653827 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.653605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxr4\" (UniqueName: \"kubernetes.io/projected/305b651b-c1bc-4b0b-99fb-091bebcebb06-kube-api-access-cwxr4\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:04.653827 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.653639 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-var-lib-kubelet\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.653827 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.653664 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-daemon-config\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.653827 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.653707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-system-cni-dir\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.653827 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.653721 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:04.653827 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.653784 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-system-cni-dir\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.653827 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:04.653817 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs podName:305b651b-c1bc-4b0b-99fb-091bebcebb06 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:05.15379233 +0000 UTC m=+3.099377685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs") pod "network-metrics-daemon-vsq4t" (UID: "305b651b-c1bc-4b0b-99fb-091bebcebb06") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:04.654204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.653836 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-var-lib-kubelet\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.653973 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-run-multus-certs\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94rw\" (UniqueName: \"kubernetes.io/projected/02c4a893-1ad4-4389-81b7-9bb7ded935b6-kube-api-access-q94rw\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654062 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2x6\" (UniqueName: \"kubernetes.io/projected/c4cb04e9-3bdd-4c7b-8b9f-8063c3605454-kube-api-access-dr2x6\") pod \"node-resolver-5tmhb\" (UID: \"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454\") " pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.654204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654068 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-run-multus-certs\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654091 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-run-k8s-cni-cncf-io\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654130 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-run-netns\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-conf-dir\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-cnibin\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-etc-kubernetes\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4cb04e9-3bdd-4c7b-8b9f-8063c3605454-hosts-file\") pod \"node-resolver-5tmhb\" (UID: \"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454\") " pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654302 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-cni-dir\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654323 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-run-netns\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654326 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-os-release\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02c4a893-1ad4-4389-81b7-9bb7ded935b6-cni-binary-copy\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654395 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-os-release\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654403 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-socket-dir-parent\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654412 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-daemon-config\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654436 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-var-lib-cni-multus\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654458 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-etc-kubernetes\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654460 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-socket-dir-parent\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654462 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-hostroot\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654499 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-cnibin\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654497 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4cb04e9-3bdd-4c7b-8b9f-8063c3605454-tmp-dir\") pod \"node-resolver-5tmhb\" (UID: \"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454\") " pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654525 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-var-lib-cni-bin\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654540 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4cb04e9-3bdd-4c7b-8b9f-8063c3605454-hosts-file\") pod \"node-resolver-5tmhb\" (UID: \"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454\") " pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654498 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-hostroot\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.654683 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654642 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-cni-dir\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.655538 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654672 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-var-lib-cni-bin\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.655538 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654677 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-var-lib-cni-multus\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.655538 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654708 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-host-run-k8s-cni-cncf-io\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.655538 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654807 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4cb04e9-3bdd-4c7b-8b9f-8063c3605454-tmp-dir\") pod \"node-resolver-5tmhb\" (UID: \"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454\") " pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.655538 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.654866 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02c4a893-1ad4-4389-81b7-9bb7ded935b6-multus-conf-dir\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.655538 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.655213 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02c4a893-1ad4-4389-81b7-9bb7ded935b6-cni-binary-copy\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.667650 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.667578 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94rw\" (UniqueName: \"kubernetes.io/projected/02c4a893-1ad4-4389-81b7-9bb7ded935b6-kube-api-access-q94rw\") pod \"multus-jmpjk\" (UID: \"02c4a893-1ad4-4389-81b7-9bb7ded935b6\") " pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.667873 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.667851 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxr4\" (UniqueName: \"kubernetes.io/projected/305b651b-c1bc-4b0b-99fb-091bebcebb06-kube-api-access-cwxr4\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:04.667925 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.667878 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2x6\" (UniqueName: \"kubernetes.io/projected/c4cb04e9-3bdd-4c7b-8b9f-8063c3605454-kube-api-access-dr2x6\") pod \"node-resolver-5tmhb\" (UID: \"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454\") " pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.754288 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.754247 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82nrx" Jul 06 23:56:04.763777 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.763751 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:04.770751 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.770728 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jrnh" Jul 06 23:56:04.778714 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.778687 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v49gd" Jul 06 23:56:04.786540 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.786515 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" Jul 06 23:56:04.796743 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.796705 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zrxrv" Jul 06 23:56:04.804904 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.804864 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:04.812645 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.812621 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5tmhb" Jul 06 23:56:04.821971 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.821787 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jmpjk" Jul 06 23:56:04.932475 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:04.932386 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:56:05.057848 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:05.057808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:05.058033 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:05.058006 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:05.058113 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:05.058098 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret podName:f1d07f90-5abf-453f-bc2a-920be5e548a8 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:06.058076022 +0000 UTC m=+4.003661373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret") pod "global-pull-secret-syncer-4qtnq" (UID: "f1d07f90-5abf-453f-bc2a-920be5e548a8") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:05.158924 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:05.158881 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:05.159126 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:05.158993 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:05.159126 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:05.159068 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:05.159126 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:05.159096 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:05.159126 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:05.159107 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xm59l for pod openshift-network-diagnostics/network-check-target-mkjcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:05.159126 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:05.159113 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:05.159351 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:05.159166 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l podName:612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc nodeName:}" failed. No retries permitted until 2026-07-06 23:56:06.159151233 +0000 UTC m=+4.104736583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xm59l" (UniqueName: "kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l") pod "network-check-target-mkjcg" (UID: "612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:05.159351 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:05.159186 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs podName:305b651b-c1bc-4b0b-99fb-091bebcebb06 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:06.159176927 +0000 UTC m=+4.104762276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs") pod "network-metrics-daemon-vsq4t" (UID: "305b651b-c1bc-4b0b-99fb-091bebcebb06") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:05.480696 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:05.480656 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:51:04 +0000 UTC" deadline="2028-02-10 01:13:59.041961104 +0000 UTC" Jul 06 23:56:05.480696 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:05.480689 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13993h17m53.561277965s" Jul 06 23:56:05.571065 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:05.570980 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:05.571236 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:05.571118 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:05.615035 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:05.615002 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c4a893_1ad4_4389_81b7_9bb7ded935b6.slice/crio-cccaa4cfc820437c99b42a4fbaa6444edc6ca5d9061f7e2b174bee095bf57a1d WatchSource:0}: Error finding container cccaa4cfc820437c99b42a4fbaa6444edc6ca5d9061f7e2b174bee095bf57a1d: Status 404 returned error can't find the container with id cccaa4cfc820437c99b42a4fbaa6444edc6ca5d9061f7e2b174bee095bf57a1d Jul 06 23:56:05.616804 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:05.616776 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf955c1d2_b48b_4ac9_aa50_bdff45fd5813.slice/crio-a66a873e02a6724c88de8e1da56106ac2d39e959a761d1884d0cf6351668aaff WatchSource:0}: Error finding container a66a873e02a6724c88de8e1da56106ac2d39e959a761d1884d0cf6351668aaff: Status 404 returned error can't find the container with id a66a873e02a6724c88de8e1da56106ac2d39e959a761d1884d0cf6351668aaff Jul 06 23:56:05.617687 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:05.617650 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43576a49_39c7_4f5e_9537_792cc49db45c.slice/crio-500c8bacb2c96dfeacc3d2e1b9a9a5df3788845df35988cacedc7c76843f0146 WatchSource:0}: Error finding container 500c8bacb2c96dfeacc3d2e1b9a9a5df3788845df35988cacedc7c76843f0146: Status 404 returned error can't find the container with id 500c8bacb2c96dfeacc3d2e1b9a9a5df3788845df35988cacedc7c76843f0146 Jul 06 23:56:05.619627 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:05.619460 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cb04e9_3bdd_4c7b_8b9f_8063c3605454.slice/crio-55cba843df67845c0beaecc2b4d09e15a0006bbd5139e39557d276f840b6feae WatchSource:0}: Error finding container 55cba843df67845c0beaecc2b4d09e15a0006bbd5139e39557d276f840b6feae: Status 404 returned error can't find the container with id 55cba843df67845c0beaecc2b4d09e15a0006bbd5139e39557d276f840b6feae Jul 06 23:56:05.621930 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:05.621903 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafda7b21_8a0a_440c_bf34_3dd094197ca7.slice/crio-2397f84bf927f799b01ebf989dc7859370697ff1523004085efe050165341064 WatchSource:0}: Error finding container 2397f84bf927f799b01ebf989dc7859370697ff1523004085efe050165341064: Status 404 returned error can't find the container with id 2397f84bf927f799b01ebf989dc7859370697ff1523004085efe050165341064 Jul 06 23:56:05.622523 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:05.622498 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c62bce_e3c3_4979_ac9e_5c3d187870df.slice/crio-560ce304c6f6ed9088c8d8492907396d9574f567b6e171eb5ff1dba31beb4288 WatchSource:0}: Error finding container 560ce304c6f6ed9088c8d8492907396d9574f567b6e171eb5ff1dba31beb4288: Status 404 returned error can't find the container with id 560ce304c6f6ed9088c8d8492907396d9574f567b6e171eb5ff1dba31beb4288 Jul 06 23:56:06.067613 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.066751 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:06.067613 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.066951 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:06.067613 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.067176 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret podName:f1d07f90-5abf-453f-bc2a-920be5e548a8 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:08.067150201 +0000 UTC m=+6.012735577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret") pod "global-pull-secret-syncer-4qtnq" (UID: "f1d07f90-5abf-453f-bc2a-920be5e548a8") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:06.173401 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.168196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:06.173401 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.168271 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:06.173401 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.168434 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:06.173401 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.168455 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:06.173401 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.168469 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xm59l for pod openshift-network-diagnostics/network-check-target-mkjcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:06.173401 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.168536 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l podName:612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc nodeName:}" failed. No retries permitted until 2026-07-06 23:56:08.168515295 +0000 UTC m=+6.114100651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xm59l" (UniqueName: "kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l") pod "network-check-target-mkjcg" (UID: "612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:06.173401 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.168993 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:06.173401 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.169056 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs podName:305b651b-c1bc-4b0b-99fb-091bebcebb06 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:08.169038262 +0000 UTC m=+6.114623619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs") pod "network-metrics-daemon-vsq4t" (UID: "305b651b-c1bc-4b0b-99fb-091bebcebb06") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:06.481504 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.481232 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:51:04 +0000 UTC" deadline="2028-01-09 09:22:58.534723365 +0000 UTC" Jul 06 23:56:06.481504 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.481379 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13233h26m52.053349034s" Jul 06 23:56:06.572372 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.571649 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:06.572372 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.571811 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:06.572372 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.572220 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:06.572372 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:06.572329 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:06.598498 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.598463 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" event={"ID":"f955c1d2-b48b-4ac9-aa50-bdff45fd5813","Type":"ContainerStarted","Data":"a66a873e02a6724c88de8e1da56106ac2d39e959a761d1884d0cf6351668aaff"} Jul 06 23:56:06.609921 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.609882 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jmpjk" event={"ID":"02c4a893-1ad4-4389-81b7-9bb7ded935b6","Type":"ContainerStarted","Data":"cccaa4cfc820437c99b42a4fbaa6444edc6ca5d9061f7e2b174bee095bf57a1d"} Jul 06 23:56:06.621413 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.621368 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jrnh" event={"ID":"aa00bed5-0833-4872-99cb-7c9d933d9323","Type":"ContainerStarted","Data":"13e8dbb8317c2a5ef33d5ef6bc01e89deb9f613852b680f48234a4bba9f04876"} Jul 06 23:56:06.634995 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.634943 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zrxrv" event={"ID":"43576a49-39c7-4f5e-9537-792cc49db45c","Type":"ContainerStarted","Data":"500c8bacb2c96dfeacc3d2e1b9a9a5df3788845df35988cacedc7c76843f0146"} Jul 06 23:56:06.643969 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.643921 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82nrx" event={"ID":"ff834cf2-cbcd-4722-9c4e-597f34f52036","Type":"ContainerStarted","Data":"4d8663481f2ffda8a738372715b0dd6015402ac5b67529aa14ff771ad1516edb"} Jul 06 23:56:06.650481 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.650442 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal" event={"ID":"d45b6b823d2fa18cb79d62947f7a1859","Type":"ContainerStarted","Data":"cd866a5a75e8837fd674a4fb00452ea39f97a112d04bd814c90e9719d41f0f83"} Jul 06 23:56:06.654568 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.654516 2563 generic.go:356] "Generic (PLEG): container finished" podID="ef76d5125552ec29a9aa27d158114806" containerID="9169b563be01b9afc02f4938fe9b5d3c7c5dbe220173743dda22a4cc41e093a3" exitCode=0 Jul 06 23:56:06.655185 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.655158 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" event={"ID":"ef76d5125552ec29a9aa27d158114806","Type":"ContainerDied","Data":"9169b563be01b9afc02f4938fe9b5d3c7c5dbe220173743dda22a4cc41e093a3"} Jul 06 23:56:06.673304 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.673270 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" event={"ID":"afda7b21-8a0a-440c-bf34-3dd094197ca7","Type":"ContainerStarted","Data":"2397f84bf927f799b01ebf989dc7859370697ff1523004085efe050165341064"} Jul 06 23:56:06.681177 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.681109 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.ec2.internal" podStartSLOduration=2.681084952 podStartE2EDuration="2.681084952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:06.666340425 +0000 UTC m=+4.611925801" watchObservedRunningTime="2026-07-06 23:56:06.681084952 +0000 UTC m=+4.626670525" Jul 06 23:56:06.687977 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.687935 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5tmhb" event={"ID":"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454","Type":"ContainerStarted","Data":"55cba843df67845c0beaecc2b4d09e15a0006bbd5139e39557d276f840b6feae"} Jul 06 23:56:06.706104 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.706057 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4rfjl" event={"ID":"99c62bce-e3c3-4979-ac9e-5c3d187870df","Type":"ContainerStarted","Data":"560ce304c6f6ed9088c8d8492907396d9574f567b6e171eb5ff1dba31beb4288"} Jul 06 23:56:06.726045 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:06.725993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v49gd" event={"ID":"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec","Type":"ContainerStarted","Data":"5c4d929f44a45ef8c3736f42f5d5c9374ca9105a980b435e762d071196b18ab2"} Jul 06 23:56:07.570676 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:07.570638 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:07.571204 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:07.570779 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:07.747709 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:07.747668 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" event={"ID":"ef76d5125552ec29a9aa27d158114806","Type":"ContainerStarted","Data":"7dc3307752283fa46a918b1c17011752b7a1fbbca1d37d7bf30296e23bae8c8f"} Jul 06 23:56:08.095653 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:08.095590 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:08.095856 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.095783 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:08.095926 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.095857 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret podName:f1d07f90-5abf-453f-bc2a-920be5e548a8 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:12.095835503 +0000 UTC m=+10.041420861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret") pod "global-pull-secret-syncer-4qtnq" (UID: "f1d07f90-5abf-453f-bc2a-920be5e548a8") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:08.197648 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:08.196711 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:08.197648 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:08.196778 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:08.197648 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.196964 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:08.197648 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.196984 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:08.197648 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.197000 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xm59l for pod openshift-network-diagnostics/network-check-target-mkjcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:08.197648 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.197067 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l podName:612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc nodeName:}" failed. No retries permitted until 2026-07-06 23:56:12.197047543 +0000 UTC m=+10.142632913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xm59l" (UniqueName: "kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l") pod "network-check-target-mkjcg" (UID: "612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:08.197648 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.197507 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:08.197648 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.197561 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs podName:305b651b-c1bc-4b0b-99fb-091bebcebb06 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:12.197543259 +0000 UTC m=+10.143128631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs") pod "network-metrics-daemon-vsq4t" (UID: "305b651b-c1bc-4b0b-99fb-091bebcebb06") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:08.571797 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:08.570967 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:08.571797 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.571130 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:08.571797 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:08.570973 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:08.571797 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:08.571660 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:09.570952 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:09.570908 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:09.571241 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:09.571061 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:10.572245 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:10.571723 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:10.572245 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:10.571873 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:10.572245 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:10.571935 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:10.572245 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:10.572056 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:11.570811 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:11.570773 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:11.570996 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:11.570919 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:12.130043 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:12.129427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:12.130043 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.129606 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:12.130043 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.129671 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret podName:f1d07f90-5abf-453f-bc2a-920be5e548a8 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:20.129652299 +0000 UTC m=+18.075237651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret") pod "global-pull-secret-syncer-4qtnq" (UID: "f1d07f90-5abf-453f-bc2a-920be5e548a8") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:12.230304 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:12.230261 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:12.230505 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:12.230337 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:12.230505 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.230475 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:12.230505 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.230486 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:12.230505 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.230501 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:12.230723 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.230513 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xm59l for pod openshift-network-diagnostics/network-check-target-mkjcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:12.230723 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.230557 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs podName:305b651b-c1bc-4b0b-99fb-091bebcebb06 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:20.230521184 +0000 UTC m=+18.176106556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs") pod "network-metrics-daemon-vsq4t" (UID: "305b651b-c1bc-4b0b-99fb-091bebcebb06") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:12.230723 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.230577 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l podName:612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc nodeName:}" failed. No retries permitted until 2026-07-06 23:56:20.230568205 +0000 UTC m=+18.176153561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xm59l" (UniqueName: "kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l") pod "network-check-target-mkjcg" (UID: "612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:12.572243 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:12.572156 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:12.572393 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.572283 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:12.572736 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:12.572714 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:12.572857 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:12.572834 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:13.571144 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:13.570712 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:13.571144 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:13.570829 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:14.571204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:14.570648 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:14.571204 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:14.570681 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:14.571204 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:14.570786 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:14.571204 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:14.570931 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:15.570466 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:15.570434 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:15.570649 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:15.570568 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:16.570693 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:16.570657 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:16.571053 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:16.570666 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:16.571053 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:16.570779 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:16.571053 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:16.570853 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:17.570712 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:17.570667 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:17.571141 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:17.570802 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:18.571107 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:18.571064 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:18.571599 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:18.571067 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:18.571599 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:18.571200 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:18.571599 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:18.571310 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:19.571187 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:19.571153 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:19.571740 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:19.571273 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:20.190472 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:20.190437 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:20.190688 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.190593 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:20.190688 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.190662 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret podName:f1d07f90-5abf-453f-bc2a-920be5e548a8 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:36.19064407 +0000 UTC m=+34.136229419 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret") pod "global-pull-secret-syncer-4qtnq" (UID: "f1d07f90-5abf-453f-bc2a-920be5e548a8") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:20.291042 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:20.291005 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:20.291227 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:20.291093 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:20.291227 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.291193 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:20.291227 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.291207 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:20.291227 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.291218 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:20.291227 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.291228 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xm59l for pod openshift-network-diagnostics/network-check-target-mkjcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:20.291487 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.291267 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs podName:305b651b-c1bc-4b0b-99fb-091bebcebb06 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:36.291252231 +0000 UTC m=+34.236837580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs") pod "network-metrics-daemon-vsq4t" (UID: "305b651b-c1bc-4b0b-99fb-091bebcebb06") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:20.291487 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.291287 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l podName:612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc nodeName:}" failed. No retries permitted until 2026-07-06 23:56:36.291276801 +0000 UTC m=+34.236862157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xm59l" (UniqueName: "kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l") pod "network-check-target-mkjcg" (UID: "612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:20.570813 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:20.570714 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:20.570982 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.570855 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:20.570982 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:20.570896 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:20.571088 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:20.571022 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:21.570475 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:21.570435 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:21.570917 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:21.570573 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:22.571401 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:22.571355 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:22.571948 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:22.571488 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:22.571948 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:22.571519 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:22.571948 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:22.571648 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:23.570548 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:23.570500 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:23.570725 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:23.570629 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:24.570762 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:24.570725 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:24.571274 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:24.570738 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:24.571274 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:24.570889 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:24.571274 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:24.570941 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:25.571135 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.570924 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:25.571964 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:25.571253 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:25.794967 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.794844 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" event={"ID":"afda7b21-8a0a-440c-bf34-3dd094197ca7","Type":"ContainerStarted","Data":"d08a5caa33d962e9ca63653f9851d6d35bb4250024ffc0ea576ae527e4766a4a"} Jul 06 23:56:25.794967 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.794944 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" event={"ID":"afda7b21-8a0a-440c-bf34-3dd094197ca7","Type":"ContainerStarted","Data":"cf5875d58b66f9927578b96a6008e275cdc0a4b2f3b95407abbd68a102015691"} Jul 06 23:56:25.795189 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.794993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" event={"ID":"afda7b21-8a0a-440c-bf34-3dd094197ca7","Type":"ContainerStarted","Data":"f9efdef831e4e3140ebfa8acf82f7f44ba28fe0132e0324c07914ebe4cf2ee0a"} Jul 06 23:56:25.795189 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.795029 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" event={"ID":"afda7b21-8a0a-440c-bf34-3dd094197ca7","Type":"ContainerStarted","Data":"66d9d7a3443f9fe9fe1ff2d0c854924a8f9a41911a04f77065163c292e60da69"} Jul 06 23:56:25.795189 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.795075 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" event={"ID":"afda7b21-8a0a-440c-bf34-3dd094197ca7","Type":"ContainerStarted","Data":"58dc090bd30a5bf4211651531873ed87e72aadc761683d572f9fa7294aa939d9"} Jul 06 23:56:25.795189 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.795106 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" event={"ID":"afda7b21-8a0a-440c-bf34-3dd094197ca7","Type":"ContainerStarted","Data":"5dec0eae6d8e20848f963599b5f2b25e3e3abe670147dec4289759a6fef2936e"} Jul 06 23:56:25.796498 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.796451 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5tmhb" event={"ID":"c4cb04e9-3bdd-4c7b-8b9f-8063c3605454","Type":"ContainerStarted","Data":"9b78ec82f9c16bb1c10f7f3bb5abc4122c7f394ef315a43450b77393c720429a"} Jul 06 23:56:25.797890 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.797869 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4rfjl" event={"ID":"99c62bce-e3c3-4979-ac9e-5c3d187870df","Type":"ContainerStarted","Data":"ea665607c8b8da89095eb89d9063a7cf2e93f695ec252fff8ec5aa82cb6f0d6b"} Jul 06 23:56:25.799495 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.799470 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v49gd" event={"ID":"c4d90ac1-1f6b-451c-85cd-5d9c30acfcec","Type":"ContainerStarted","Data":"96fb1334d1e68744ce4acf1094071ab3e0e7be41daae04013ae0745319a22798"} Jul 06 23:56:25.801653 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.801619 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" event={"ID":"f955c1d2-b48b-4ac9-aa50-bdff45fd5813","Type":"ContainerStarted","Data":"216c6572742b5f5ff2760c97193ab6e77567f137719f4d0f823f46230ac4b828"} Jul 06 23:56:25.803274 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.803233 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jmpjk" event={"ID":"02c4a893-1ad4-4389-81b7-9bb7ded935b6","Type":"ContainerStarted","Data":"75cece628c1140d22d6a643117bd687a16636960c4018caed2bedef9f097130f"} Jul 06 23:56:25.805083 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.805060 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jrnh" event={"ID":"aa00bed5-0833-4872-99cb-7c9d933d9323","Type":"ContainerStarted","Data":"d81730ec8e8ba089999e568d1baf4b34c514ebb562edaf0a0b0132078c2ad6c8"} Jul 06 23:56:25.806788 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.806765 2563 generic.go:356] "Generic (PLEG): container finished" podID="ff834cf2-cbcd-4722-9c4e-597f34f52036" containerID="623282a2f7d2c8e60f459bc97c00f342ce451d3209549392e091fd4ab632004a" exitCode=0 Jul 06 23:56:25.806876 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.806807 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82nrx" event={"ID":"ff834cf2-cbcd-4722-9c4e-597f34f52036","Type":"ContainerDied","Data":"623282a2f7d2c8e60f459bc97c00f342ce451d3209549392e091fd4ab632004a"} Jul 06 23:56:25.816270 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.816163 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.ec2.internal" podStartSLOduration=21.816143175 podStartE2EDuration="21.816143175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:07.766310703 +0000 UTC m=+5.711896079" watchObservedRunningTime="2026-07-06 23:56:25.816143175 +0000 UTC m=+23.761728546" Jul 06 23:56:25.817044 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.817000 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5tmhb" podStartSLOduration=3.482969485 podStartE2EDuration="22.816977451s" totalImagesPullingTime="19.334007966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:25.815736696 +0000 UTC m=+23.761322058" watchObservedRunningTime="2026-07-06 23:56:25.816977451 +0000 UTC m=+23.762562825" Jul 06 23:56:25.897462 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.897403 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8jrnh" podStartSLOduration=3.571965303 podStartE2EDuration="22.897382s" totalImagesPullingTime="19.325416697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:25.896770951 +0000 UTC m=+23.842356323" watchObservedRunningTime="2026-07-06 23:56:25.897382 +0000 UTC m=+23.842967372" Jul 06 23:56:25.918671 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.918625 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v49gd" podStartSLOduration=3.5798503 podStartE2EDuration="22.918608153s" totalImagesPullingTime="19.338757853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:25.918508976 +0000 UTC m=+23.864094349" watchObservedRunningTime="2026-07-06 23:56:25.918608153 +0000 UTC m=+23.864193533" Jul 06 23:56:25.941341 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.941283 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4rfjl" podStartSLOduration=11.614846316 podStartE2EDuration="22.941268633s" totalImagesPullingTime="11.326422317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:25.941008124 +0000 UTC m=+23.886593495" watchObservedRunningTime="2026-07-06 23:56:25.941268633 +0000 UTC m=+23.886854003" Jul 06 23:56:25.959045 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.958996 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jmpjk" podStartSLOduration=3.559650517 podStartE2EDuration="22.958983671s" totalImagesPullingTime="19.399333154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:25.958842169 +0000 UTC m=+23.904427541" watchObservedRunningTime="2026-07-06 23:56:25.958983671 +0000 UTC m=+23.904569079" Jul 06 23:56:25.968803 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:25.968771 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:26.139669 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.139588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:26.391263 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.391187 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 23:56:26.504634 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.504500 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T23:56:26.391210081Z","UUID":"4a03c8f2-c8a8-4546-a803-3e9b48e23e89","Handler":null,"Name":"","Endpoint":""} Jul 06 23:56:26.506110 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.506083 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 23:56:26.506253 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.506128 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 23:56:26.570502 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.570475 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:26.570502 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.570516 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:26.570769 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:26.570646 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:26.570825 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:26.570801 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:26.748931 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.748893 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:26.811016 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.810981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" event={"ID":"f955c1d2-b48b-4ac9-aa50-bdff45fd5813","Type":"ContainerStarted","Data":"b4c699663b7615d4a0901bd7b00e151a4b03f4f2bac545aac4dc0cfdbd23d0ed"} Jul 06 23:56:26.812674 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:26.812641 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zrxrv" event={"ID":"43576a49-39c7-4f5e-9537-792cc49db45c","Type":"ContainerStarted","Data":"370c282b470563aff4a999fa8b07f695841b2cc90557072c42876bc0ce8dabcd"} Jul 06 23:56:27.141130 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:27.141044 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:27.540691 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:27.540657 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:27.571014 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:27.570986 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:27.571164 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:27.571108 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:27.757330 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:27.757299 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:27.817285 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:27.817244 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" event={"ID":"f955c1d2-b48b-4ac9-aa50-bdff45fd5813","Type":"ContainerStarted","Data":"946eca8f70529c8ad966d09ea6d381caf07e78f161fd7bff92034608c91cc93c"} Jul 06 23:56:27.836499 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:27.836442 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lm97k" podStartSLOduration=3.247738251 podStartE2EDuration="24.836428095s" totalImagesPullingTime="21.588689844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:27.83605926 +0000 UTC m=+25.781644632" watchObservedRunningTime="2026-07-06 23:56:27.836428095 +0000 UTC m=+25.782013472" Jul 06 23:56:27.836761 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:27.836700 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zrxrv" podStartSLOduration=5.50097469 podStartE2EDuration="24.836694312s" totalImagesPullingTime="19.335719622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:26.827913049 +0000 UTC m=+24.773498421" watchObservedRunningTime="2026-07-06 23:56:27.836694312 +0000 UTC m=+25.782279682" Jul 06 23:56:27.939824 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:27.939795 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:28.004094 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:28.004058 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:28.004766 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:28.004742 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:28.145285 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:28.145250 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:28.571297 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:28.570750 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:28.571297 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:28.570760 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:28.571297 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:28.570907 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:28.571297 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:28.570984 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:28.823514 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:28.823419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" event={"ID":"afda7b21-8a0a-440c-bf34-3dd094197ca7","Type":"ContainerStarted","Data":"ae80ddba941f17d23263f47343b78cf96976e03c5be7cf02fd1966c7572db2f6"} Jul 06 23:56:28.823984 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:28.823849 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:28.824266 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:28.824239 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4rfjl" Jul 06 23:56:29.044174 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:29.044139 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:29.143598 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:29.143492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:29.244924 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:29.244894 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:29.344598 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:29.344509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:29.571091 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:29.571051 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:29.571262 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:29.571188 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:29.782942 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:29.782911 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:29.886672 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:29.886576 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:29.981504 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:29.981462 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:30.331568 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:30.331378 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:30.570613 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:30.570584 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:30.570781 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:30.570717 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:30.570781 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:30.570771 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:30.570924 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:30.570900 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:30.832002 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:30.831901 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" event={"ID":"afda7b21-8a0a-440c-bf34-3dd094197ca7","Type":"ContainerStarted","Data":"b8f34eed11940c97cbd2136349d66dbae6e2c95f448c7ad3709bdcc9a6713c51"} Jul 06 23:56:30.832231 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:30.832191 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:30.834350 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:30.834326 2563 generic.go:356] "Generic (PLEG): container finished" podID="ff834cf2-cbcd-4722-9c4e-597f34f52036" containerID="18c5c088ea6517391642e74bba51b0479b0185496881f740c9cf0a432f3916bc" exitCode=0 Jul 06 23:56:30.834464 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:30.834409 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82nrx" event={"ID":"ff834cf2-cbcd-4722-9c4e-597f34f52036","Type":"ContainerDied","Data":"18c5c088ea6517391642e74bba51b0479b0185496881f740c9cf0a432f3916bc"} Jul 06 23:56:30.848591 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:30.848566 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:30.861687 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:30.861644 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" podStartSLOduration=8.451213634 podStartE2EDuration="27.861630699s" totalImagesPullingTime="19.410417065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:30.859971753 +0000 UTC m=+28.805557124" watchObservedRunningTime="2026-07-06 23:56:30.861630699 +0000 UTC m=+28.807216069" Jul 06 23:56:31.140431 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:31.140340 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:31.417697 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:31.417664 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:31.570346 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:31.570302 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:31.570522 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:31.570426 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:31.737887 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:31.737647 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:31.796008 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:31.795973 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:31.838935 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:31.838888 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:31.839341 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:31.839323 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:31.855013 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:31.854985 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:56:31.933577 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:31.933545 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:32.337725 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:32.337618 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:32.571947 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:32.571905 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:32.572142 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:32.572019 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:32.572142 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:32.572055 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:32.572142 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:32.572137 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:32.842066 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:32.842034 2563 generic.go:356] "Generic (PLEG): container finished" podID="ff834cf2-cbcd-4722-9c4e-597f34f52036" containerID="bb019fd81cb6dfcf4551896d274dddea2f8c88ffa6724a98a900c6160805e9da" exitCode=0 Jul 06 23:56:32.842274 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:32.842129 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82nrx" event={"ID":"ff834cf2-cbcd-4722-9c4e-597f34f52036","Type":"ContainerDied","Data":"bb019fd81cb6dfcf4551896d274dddea2f8c88ffa6724a98a900c6160805e9da"} Jul 06 23:56:32.944646 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:32.944608 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:33.350275 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:33.350236 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:33.532831 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:33.532803 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:33.570851 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:33.570822 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:33.571010 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:33.570943 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:34.136763 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:34.136669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:34.312465 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:34.312430 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:34.571105 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:34.571070 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:34.571566 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:34.571184 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:34.571566 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:34.571250 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:34.571566 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:34.571373 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:34.734995 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:34.734960 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:34.849718 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:34.849628 2563 generic.go:356] "Generic (PLEG): container finished" podID="ff834cf2-cbcd-4722-9c4e-597f34f52036" containerID="95e173bc6c5dcddda1ffa30facbfdcf66d10a9221e4d117f3131b94f50c90d78" exitCode=0 Jul 06 23:56:34.849718 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:34.849674 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82nrx" event={"ID":"ff834cf2-cbcd-4722-9c4e-597f34f52036","Type":"ContainerDied","Data":"95e173bc6c5dcddda1ffa30facbfdcf66d10a9221e4d117f3131b94f50c90d78"} Jul 06 23:56:34.931330 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:34.931296 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:35.133991 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:35.133908 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:35.220100 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:35.220070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:35.567370 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:35.567338 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:35.570216 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:35.570201 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:35.570310 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:35.570294 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:35.938355 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:35.938314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:36.142236 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:36.142196 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:36.206869 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:36.206787 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:36.207033 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.206958 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:36.207076 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.207043 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret podName:f1d07f90-5abf-453f-bc2a-920be5e548a8 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:08.207019524 +0000 UTC m=+66.152604891 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret") pod "global-pull-secret-syncer-4qtnq" (UID: "f1d07f90-5abf-453f-bc2a-920be5e548a8") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:56:36.307840 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:36.307808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:36.308020 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:36.307858 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:36.308020 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.307969 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:36.308113 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.308044 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs podName:305b651b-c1bc-4b0b-99fb-091bebcebb06 nodeName:}" failed. No retries permitted until 2026-07-06 23:57:08.308024316 +0000 UTC m=+66.253609679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs") pod "network-metrics-daemon-vsq4t" (UID: "305b651b-c1bc-4b0b-99fb-091bebcebb06") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:56:36.308113 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.307971 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:56:36.308113 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.308088 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:56:36.308113 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.308102 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xm59l for pod openshift-network-diagnostics/network-check-target-mkjcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:36.309619 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.309591 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l podName:612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc nodeName:}" failed. No retries permitted until 2026-07-06 23:57:08.308475578 +0000 UTC m=+66.254060972 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xm59l" (UniqueName: "kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l") pod "network-check-target-mkjcg" (UID: "612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:56:36.573720 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:36.573637 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:36.573884 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.573751 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:36.573884 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:36.573809 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:36.573996 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:36.573919 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:37.570653 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:37.570616 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:37.571051 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:37.570764 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:38.570810 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:38.570774 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:38.571196 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:38.570775 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:38.571196 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:38.570916 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:38.571196 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:38.570957 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:39.570445 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:39.570421 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:39.570582 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:39.570524 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:39.863801 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:39.863714 2563 generic.go:356] "Generic (PLEG): container finished" podID="ff834cf2-cbcd-4722-9c4e-597f34f52036" containerID="21465911156ad8ff62e9917237c770d0b5a34682fbfa20cccb567fbeabe7c484" exitCode=0 Jul 06 23:56:39.863801 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:39.863774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82nrx" event={"ID":"ff834cf2-cbcd-4722-9c4e-597f34f52036","Type":"ContainerDied","Data":"21465911156ad8ff62e9917237c770d0b5a34682fbfa20cccb567fbeabe7c484"} Jul 06 23:56:40.570853 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:40.570818 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:40.571059 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:40.570818 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:40.571059 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:40.570938 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:40.571059 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:40.571022 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:40.868947 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:40.868868 2563 generic.go:356] "Generic (PLEG): container finished" podID="ff834cf2-cbcd-4722-9c4e-597f34f52036" containerID="2c393640542a72523ce259a3386accce00507e0ac2e4249002dee392be865b27" exitCode=0 Jul 06 23:56:40.868947 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:40.868925 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82nrx" event={"ID":"ff834cf2-cbcd-4722-9c4e-597f34f52036","Type":"ContainerDied","Data":"2c393640542a72523ce259a3386accce00507e0ac2e4249002dee392be865b27"} Jul 06 23:56:41.570994 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:41.570957 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:41.571181 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:41.571058 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:41.833577 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:41.833474 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:41.874921 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:41.874889 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82nrx" event={"ID":"ff834cf2-cbcd-4722-9c4e-597f34f52036","Type":"ContainerStarted","Data":"d3071ba0a741b1f264dc5da6dd9c8e11a9e8544f17f2d2ddf4e961e90b097875"} Jul 06 23:56:41.906082 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:41.906031 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-82nrx" podStartSLOduration=8.235331107 podStartE2EDuration="38.906014746s" totalImagesPullingTime="30.670683639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:41.905811436 +0000 UTC m=+39.851396807" watchObservedRunningTime="2026-07-06 23:56:41.906014746 +0000 UTC m=+39.851600154" Jul 06 23:56:41.913403 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:41.913356 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:42.078257 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:42.078212 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:42.302642 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:42.302610 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:42.402513 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:42.402477 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:42.501443 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:42.501404 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:42.574364 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:42.574265 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:42.574507 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:42.574370 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:42.574507 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:42.574406 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:42.574507 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:42.574468 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:42.707805 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:42.707770 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:43.571118 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:43.571087 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:43.571494 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:43.571181 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:43.712873 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:43.712841 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:43.895415 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:43.895379 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:44.038867 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:44.038838 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:44.506984 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:44.506951 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:44.567247 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:44.567221 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:44.571325 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:44.571305 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:44.571665 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:44.571411 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:44.571665 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:44.571476 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:44.571665 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:44.571582 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:45.570705 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:45.570665 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:45.570901 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:45.570797 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:45.754417 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:45.754388 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:45.907822 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:45.907788 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:46.093364 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:46.093335 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:46.198631 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:46.198523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:46.291858 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:46.291830 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:46.451864 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:46.451787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:46.571050 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:46.571014 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:46.571419 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:46.571389 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:46.571781 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:46.571757 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:46.571897 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:46.571875 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:47.309489 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:47.309285 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4qtnq"] Jul 06 23:56:47.309875 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:47.309593 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:47.309875 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:47.309683 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:47.313028 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:47.312999 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mkjcg"] Jul 06 23:56:47.313191 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:47.313122 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:47.313251 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:47.313214 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:47.313963 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:47.313934 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vsq4t"] Jul 06 23:56:47.314193 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:47.314163 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:47.314318 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:47.314273 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:48.571236 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:48.571198 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:48.571825 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:48.571335 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:49.246727 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:49.246692 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:49.505369 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:49.505284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:49.570791 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:49.570754 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:49.570953 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:49.570894 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:49.570998 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:49.570951 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:49.571049 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:49.571031 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:49.583826 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:49.583793 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:49.718461 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:49.718430 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:49.795319 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:49.795235 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:49.898008 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:49.897965 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:49.970637 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:49.970603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:50.448619 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:50.448584 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:50.570799 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:50.570754 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:50.570965 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:50.570883 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qtnq" podUID="f1d07f90-5abf-453f-bc2a-920be5e548a8" Jul 06 23:56:50.879919 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:50.879824 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:50.985789 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:50.985755 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:51.570749 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.570716 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:51.570951 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.570716 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:51.570951 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:51.570849 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkjcg" podUID="612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc" Jul 06 23:56:51.570951 ip-10-0-133-6 kubenswrapper[2563]: E0706 23:56:51.570904 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vsq4t" podUID="305b651b-c1bc-4b0b-99fb-091bebcebb06" Jul 06 23:56:51.646744 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.646690 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:51.896965 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.896945 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-6.ec2.internal" event="NodeReady" Jul 06 23:56:51.897405 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.897108 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 23:56:51.944051 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.944021 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84b64754fb-tdx8n"] Jul 06 23:56:51.960485 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.960463 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:51.964131 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.964098 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 23:56:51.964666 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.964644 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84b64754fb-tdx8n"] Jul 06 23:56:51.966047 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.966031 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 23:56:51.967182 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.966795 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g68bw\"" Jul 06 23:56:51.968184 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.968161 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 23:56:51.971245 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.971222 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 23:56:51.971407 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.971385 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kps6x"] Jul 06 23:56:51.985195 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.985165 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2zd2z"] Jul 06 23:56:51.985325 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.985298 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:51.990181 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.990153 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lhsfz\"" Jul 06 23:56:51.990329 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.990153 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 23:56:51.990527 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.990509 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 23:56:51.997750 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.997726 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2zd2z"] Jul 06 23:56:51.997873 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:51.997810 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2zd2z" Jul 06 23:56:52.003471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.003443 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kps6x"] Jul 06 23:56:52.004145 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.004125 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 23:56:52.005063 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.005046 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 23:56:52.005311 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.005298 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 23:56:52.015505 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.015406 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wljwx\"" Jul 06 23:56:52.090363 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.090328 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cd577"] Jul 06 23:56:52.093441 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.093415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:52.103017 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.103001 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.105809 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.105785 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 23:56:52.105964 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.105939 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-274zg\"" Jul 06 23:56:52.106057 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.106043 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 23:56:52.106146 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.105786 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 23:56:52.106206 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.106159 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cd577"] Jul 06 23:56:52.106403 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.106382 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 23:56:52.117813 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.117788 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmcx4\" (UniqueName: \"kubernetes.io/projected/22b1ce4e-ded4-4fc5-9089-48f68aebb1f0-kube-api-access-rmcx4\") pod \"ingress-canary-2zd2z\" (UID: \"22b1ce4e-ded4-4fc5-9089-48f68aebb1f0\") " pod="openshift-ingress-canary/ingress-canary-2zd2z" Jul 06 23:56:52.117950 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.117821 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/553125df-b156-4489-8db0-934fba7fa1d6-bound-sa-token\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.117950 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.117849 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/553125df-b156-4489-8db0-934fba7fa1d6-trusted-ca\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.117950 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.117889 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/553125df-b156-4489-8db0-934fba7fa1d6-ca-trust-extracted\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.117950 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.117921 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq685\" (UniqueName: \"kubernetes.io/projected/b75bb351-dbc1-47d5-bbd8-39714250e069-kube-api-access-kq685\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.117950 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.117943 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/553125df-b156-4489-8db0-934fba7fa1d6-registry-certificates\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.118157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.117962 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b75bb351-dbc1-47d5-bbd8-39714250e069-tmp-dir\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.118157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.117984 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/553125df-b156-4489-8db0-934fba7fa1d6-registry-tls\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.118157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.117998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/553125df-b156-4489-8db0-934fba7fa1d6-installation-pull-secrets\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.118157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.118053 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22b1ce4e-ded4-4fc5-9089-48f68aebb1f0-cert\") pod \"ingress-canary-2zd2z\" (UID: \"22b1ce4e-ded4-4fc5-9089-48f68aebb1f0\") " pod="openshift-ingress-canary/ingress-canary-2zd2z" Jul 06 23:56:52.118157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.118077 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvj4\" (UniqueName: \"kubernetes.io/projected/553125df-b156-4489-8db0-934fba7fa1d6-kube-api-access-dlvj4\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.118157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.118107 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b75bb351-dbc1-47d5-bbd8-39714250e069-config-volume\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.118157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.118128 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b75bb351-dbc1-47d5-bbd8-39714250e069-metrics-tls\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.118157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.118150 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/553125df-b156-4489-8db0-934fba7fa1d6-image-registry-private-configuration\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.193251 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.193168 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:52.219273 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219241 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b75bb351-dbc1-47d5-bbd8-39714250e069-metrics-tls\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.219424 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219280 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/553125df-b156-4489-8db0-934fba7fa1d6-image-registry-private-configuration\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.219424 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219402 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmcx4\" (UniqueName: \"kubernetes.io/projected/22b1ce4e-ded4-4fc5-9089-48f68aebb1f0-kube-api-access-rmcx4\") pod \"ingress-canary-2zd2z\" (UID: \"22b1ce4e-ded4-4fc5-9089-48f68aebb1f0\") " pod="openshift-ingress-canary/ingress-canary-2zd2z" Jul 06 23:56:52.219524 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219436 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/553125df-b156-4489-8db0-934fba7fa1d6-bound-sa-token\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.219524 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219458 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/553125df-b156-4489-8db0-934fba7fa1d6-trusted-ca\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.219524 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219504 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/553125df-b156-4489-8db0-934fba7fa1d6-ca-trust-extracted\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.219693 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219556 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq685\" (UniqueName: \"kubernetes.io/projected/b75bb351-dbc1-47d5-bbd8-39714250e069-kube-api-access-kq685\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.219693 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/553125df-b156-4489-8db0-934fba7fa1d6-registry-certificates\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.219693 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b75bb351-dbc1-47d5-bbd8-39714250e069-tmp-dir\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.219693 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/553125df-b156-4489-8db0-934fba7fa1d6-registry-tls\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.219693 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219678 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/553125df-b156-4489-8db0-934fba7fa1d6-installation-pull-secrets\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.219888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219724 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61f39557-e0fa-416c-ab65-0c3ef436d755-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.219888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219760 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61f39557-e0fa-416c-ab65-0c3ef436d755-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.219888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219789 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22b1ce4e-ded4-4fc5-9089-48f68aebb1f0-cert\") pod \"ingress-canary-2zd2z\" (UID: \"22b1ce4e-ded4-4fc5-9089-48f68aebb1f0\") " pod="openshift-ingress-canary/ingress-canary-2zd2z" Jul 06 23:56:52.219888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219812 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61f39557-e0fa-416c-ab65-0c3ef436d755-crio-socket\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.219888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219836 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslkc\" (UniqueName: \"kubernetes.io/projected/61f39557-e0fa-416c-ab65-0c3ef436d755-kube-api-access-xslkc\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.219888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219865 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvj4\" (UniqueName: \"kubernetes.io/projected/553125df-b156-4489-8db0-934fba7fa1d6-kube-api-access-dlvj4\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.219888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219888 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61f39557-e0fa-416c-ab65-0c3ef436d755-data-volume\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.220231 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219930 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b75bb351-dbc1-47d5-bbd8-39714250e069-config-volume\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.220231 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.219966 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/553125df-b156-4489-8db0-934fba7fa1d6-ca-trust-extracted\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.220231 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.220061 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b75bb351-dbc1-47d5-bbd8-39714250e069-tmp-dir\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.220950 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.220504 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b75bb351-dbc1-47d5-bbd8-39714250e069-config-volume\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.220950 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.220651 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/553125df-b156-4489-8db0-934fba7fa1d6-registry-certificates\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.221509 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.221443 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/553125df-b156-4489-8db0-934fba7fa1d6-trusted-ca\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.223879 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.223856 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/553125df-b156-4489-8db0-934fba7fa1d6-installation-pull-secrets\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.223963 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.223856 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/553125df-b156-4489-8db0-934fba7fa1d6-image-registry-private-configuration\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.223963 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.223903 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b75bb351-dbc1-47d5-bbd8-39714250e069-metrics-tls\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.224045 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.223959 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22b1ce4e-ded4-4fc5-9089-48f68aebb1f0-cert\") pod \"ingress-canary-2zd2z\" (UID: \"22b1ce4e-ded4-4fc5-9089-48f68aebb1f0\") " pod="openshift-ingress-canary/ingress-canary-2zd2z" Jul 06 23:56:52.224045 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.223983 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/553125df-b156-4489-8db0-934fba7fa1d6-registry-tls\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.231632 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.231609 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmcx4\" (UniqueName: \"kubernetes.io/projected/22b1ce4e-ded4-4fc5-9089-48f68aebb1f0-kube-api-access-rmcx4\") pod \"ingress-canary-2zd2z\" (UID: \"22b1ce4e-ded4-4fc5-9089-48f68aebb1f0\") " pod="openshift-ingress-canary/ingress-canary-2zd2z" Jul 06 23:56:52.238921 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.238892 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/553125df-b156-4489-8db0-934fba7fa1d6-bound-sa-token\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.239049 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.239033 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvj4\" (UniqueName: \"kubernetes.io/projected/553125df-b156-4489-8db0-934fba7fa1d6-kube-api-access-dlvj4\") pod \"image-registry-84b64754fb-tdx8n\" (UID: \"553125df-b156-4489-8db0-934fba7fa1d6\") " pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.243421 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.243398 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq685\" (UniqueName: \"kubernetes.io/projected/b75bb351-dbc1-47d5-bbd8-39714250e069-kube-api-access-kq685\") pod \"dns-default-kps6x\" (UID: \"b75bb351-dbc1-47d5-bbd8-39714250e069\") " pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.281421 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.281381 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.302085 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.302059 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:52.306997 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.306959 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:52.308926 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.308907 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2zd2z" Jul 06 23:56:52.320891 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.320859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61f39557-e0fa-416c-ab65-0c3ef436d755-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.321019 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.320919 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61f39557-e0fa-416c-ab65-0c3ef436d755-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.321019 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.320949 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61f39557-e0fa-416c-ab65-0c3ef436d755-crio-socket\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.321019 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.320972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslkc\" (UniqueName: \"kubernetes.io/projected/61f39557-e0fa-416c-ab65-0c3ef436d755-kube-api-access-xslkc\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.321019 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.320998 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61f39557-e0fa-416c-ab65-0c3ef436d755-data-volume\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.321440 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.321413 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61f39557-e0fa-416c-ab65-0c3ef436d755-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.321570 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.321425 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61f39557-e0fa-416c-ab65-0c3ef436d755-crio-socket\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.321570 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.321414 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61f39557-e0fa-416c-ab65-0c3ef436d755-data-volume\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.323950 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.323926 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61f39557-e0fa-416c-ab65-0c3ef436d755-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.331654 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.331626 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslkc\" (UniqueName: \"kubernetes.io/projected/61f39557-e0fa-416c-ab65-0c3ef436d755-kube-api-access-xslkc\") pod \"insights-runtime-extractor-cd577\" (UID: \"61f39557-e0fa-416c-ab65-0c3ef436d755\") " pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.413059 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.412328 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cd577" Jul 06 23:56:52.490915 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.490879 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2zd2z"] Jul 06 23:56:52.494955 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.494747 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84b64754fb-tdx8n"] Jul 06 23:56:52.497263 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.497215 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kps6x"] Jul 06 23:56:52.503174 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:52.503150 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553125df_b156_4489_8db0_934fba7fa1d6.slice/crio-d2e8360c8ea2c905dd45f5dd33e451296390ea70affb631508c12d2cc924130d WatchSource:0}: Error finding container d2e8360c8ea2c905dd45f5dd33e451296390ea70affb631508c12d2cc924130d: Status 404 returned error can't find the container with id d2e8360c8ea2c905dd45f5dd33e451296390ea70affb631508c12d2cc924130d Jul 06 23:56:52.504086 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:52.504026 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75bb351_dbc1_47d5_bbd8_39714250e069.slice/crio-791aef93ece3952189cb535a78b9b8487944f5ebe68c282ca8a28a34f399e2d2 WatchSource:0}: Error finding container 791aef93ece3952189cb535a78b9b8487944f5ebe68c282ca8a28a34f399e2d2: Status 404 returned error can't find the container with id 791aef93ece3952189cb535a78b9b8487944f5ebe68c282ca8a28a34f399e2d2 Jul 06 23:56:52.573493 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.573470 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:56:52.576071 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.576042 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cd577"] Jul 06 23:56:52.578812 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.578733 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:56:52.582402 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:56:52.582375 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f39557_e0fa_416c_ab65_0c3ef436d755.slice/crio-da4722baa1935cae87d3e2ec8f3ae973261d91baef4a6d9f9b3ae5a5a0c9383f WatchSource:0}: Error finding container da4722baa1935cae87d3e2ec8f3ae973261d91baef4a6d9f9b3ae5a5a0c9383f: Status 404 returned error can't find the container with id da4722baa1935cae87d3e2ec8f3ae973261d91baef4a6d9f9b3ae5a5a0c9383f Jul 06 23:56:52.850108 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.850068 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:52.901400 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.901369 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cd577" event={"ID":"61f39557-e0fa-416c-ab65-0c3ef436d755","Type":"ContainerStarted","Data":"da4722baa1935cae87d3e2ec8f3ae973261d91baef4a6d9f9b3ae5a5a0c9383f"} Jul 06 23:56:52.902593 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.902568 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kps6x" event={"ID":"b75bb351-dbc1-47d5-bbd8-39714250e069","Type":"ContainerStarted","Data":"791aef93ece3952189cb535a78b9b8487944f5ebe68c282ca8a28a34f399e2d2"} Jul 06 23:56:52.904047 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.904019 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" event={"ID":"553125df-b156-4489-8db0-934fba7fa1d6","Type":"ContainerStarted","Data":"df334dc81952e4f22b2409c98dda8bdeb80abe8d1239563fc80cfa7895cb2ce6"} Jul 06 23:56:52.904154 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.904069 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" event={"ID":"553125df-b156-4489-8db0-934fba7fa1d6","Type":"ContainerStarted","Data":"d2e8360c8ea2c905dd45f5dd33e451296390ea70affb631508c12d2cc924130d"} Jul 06 23:56:52.904154 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.904097 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:56:52.905117 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.905093 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2zd2z" event={"ID":"22b1ce4e-ded4-4fc5-9089-48f68aebb1f0","Type":"ContainerStarted","Data":"5dab07552d7a5f128d6a30decf0443728b1c1040cc67762fd58453160a3dd242"} Jul 06 23:56:52.955375 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:52.955312 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" podStartSLOduration=7.955290109 podStartE2EDuration="7.955290109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:52.954695047 +0000 UTC m=+50.900280419" watchObservedRunningTime="2026-07-06 23:56:52.955290109 +0000 UTC m=+50.900875482" Jul 06 23:56:53.001669 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.001630 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:53.113697 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.113606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:53.207175 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.207136 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:53.308913 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.308878 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:53.570849 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.570816 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:56:53.571285 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.571262 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:56:53.576073 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.576047 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:56:53.576497 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.576471 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:56:53.577219 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.577198 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-czj4k\"" Jul 06 23:56:53.577434 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.577415 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dplr9\"" Jul 06 23:56:53.577752 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.577734 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:56:53.686770 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.686694 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:53.759185 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:53.759150 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:54.449753 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:54.449717 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:54.542554 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:54.542499 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:54.710251 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:54.710162 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:54.770926 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:54.770883 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:54.842239 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:54.842202 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:54.910949 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:54.910919 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:55.428550 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:55.428506 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:55.557180 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:55.557146 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:56.951179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:56.951146 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:57.044016 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:57.043979 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:57.175896 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:57.175860 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:57.251253 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:57.251214 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:57.515058 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:57.514984 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:57.598207 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:57.598172 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:58.122179 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:58.122129 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:58.898741 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:58.898561 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:56:58.921975 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:58.921941 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cd577" event={"ID":"61f39557-e0fa-416c-ab65-0c3ef436d755","Type":"ContainerStarted","Data":"26b4d73ad488abf6e175cef713e72f7a79ad4607a9cfc3edaf4caba6cee41479"} Jul 06 23:56:58.923554 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:58.923512 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kps6x" event={"ID":"b75bb351-dbc1-47d5-bbd8-39714250e069","Type":"ContainerStarted","Data":"2f1a4ef6d776e7eac4870ff1d0a92451ef34ad16e3479351a52f6851b2743590"} Jul 06 23:56:58.923696 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:58.923574 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kps6x" event={"ID":"b75bb351-dbc1-47d5-bbd8-39714250e069","Type":"ContainerStarted","Data":"7b99647fa48b411fb90e0bbb761c3a4287fa4a04968a6d738613ff5704ac5fcf"} Jul 06 23:56:58.923696 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:58.923603 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kps6x" Jul 06 23:56:58.924898 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:58.924873 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2zd2z" event={"ID":"22b1ce4e-ded4-4fc5-9089-48f68aebb1f0","Type":"ContainerStarted","Data":"e182430031e05fe06724535366f03d67ece6b91466b2ab360839e71cd5c016b2"} Jul 06 23:56:58.944055 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:58.944012 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kps6x" podStartSLOduration=1.88118168 podStartE2EDuration="7.943997211s" totalImagesPullingTime="6.062815531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:58.942901686 +0000 UTC m=+56.888487077" watchObservedRunningTime="2026-07-06 23:56:58.943997211 +0000 UTC m=+56.889582559" Jul 06 23:56:58.960470 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:58.960419 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2zd2z" podStartSLOduration=1.8713323 podStartE2EDuration="7.960404969s" totalImagesPullingTime="6.089072669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:58.959881851 +0000 UTC m=+56.905467234" watchObservedRunningTime="2026-07-06 23:56:58.960404969 +0000 UTC m=+56.905990348" Jul 06 23:56:59.935435 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:59.934961 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cd577" event={"ID":"61f39557-e0fa-416c-ab65-0c3ef436d755","Type":"ContainerStarted","Data":"4ee8da6648d7dfb393c52e2ed0ad5518e7c856062b2952453de205f5adf6ffd0"} Jul 06 23:56:59.942334 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:56:59.942301 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:00.042245 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:00.042210 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:00.297571 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:00.297470 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:00.369286 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:00.369248 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:01.943317 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:01.943236 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cd577" event={"ID":"61f39557-e0fa-416c-ab65-0c3ef436d755","Type":"ContainerStarted","Data":"8abd4b43506ec636a2ed732a68f425500a71a8305d0171dc1cda3c5c19ce006f"} Jul 06 23:57:01.963759 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:01.963703 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cd577" podStartSLOduration=0.98370256 podStartE2EDuration="9.963679521s" totalImagesPullingTime="8.979976961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:01.96321176 +0000 UTC m=+59.908797130" watchObservedRunningTime="2026-07-06 23:57:01.963679521 +0000 UTC m=+59.909264893" Jul 06 23:57:03.149061 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:03.149023 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:03.220612 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:03.220580 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:03.856853 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:03.856822 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwc59" Jul 06 23:57:04.027740 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:04.027706 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:04.094686 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:04.094657 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:05.610508 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:05.610476 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:07.653209 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.653174 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:07.870452 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.870415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:07.947259 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.947175 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp"] Jul 06 23:57:07.950069 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.950049 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:07.978919 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.978882 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 23:57:07.979135 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.978890 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 23:57:07.979135 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.978892 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 23:57:07.979135 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.979125 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 23:57:07.979312 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.979142 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 23:57:07.979312 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.979260 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 23:57:07.979418 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.979349 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r98b8\"" Jul 06 23:57:07.991067 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:07.991035 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp"] Jul 06 23:57:08.007098 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.007068 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kw5nb"] Jul 06 23:57:08.008927 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.008912 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.011257 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.011234 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 23:57:08.011480 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.011466 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sslvj\"" Jul 06 23:57:08.011590 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.011464 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 23:57:08.011766 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.011751 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 23:57:08.061484 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.061451 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:08.065517 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0aabe90-1c40-4edb-8585-b5de80fdeeee-metrics-client-ca\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.065676 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065522 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qztd\" (UniqueName: \"kubernetes.io/projected/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-api-access-5qztd\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.065676 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065555 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-tls\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.065676 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.065676 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065591 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-textfile\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.065676 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065607 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c0aabe90-1c40-4edb-8585-b5de80fdeeee-root\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.065676 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065624 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-wtmp\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.065676 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.065676 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065658 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zrt4\" (UniqueName: \"kubernetes.io/projected/c0aabe90-1c40-4edb-8585-b5de80fdeeee-kube-api-access-2zrt4\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.065946 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065727 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.065946 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065750 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.065946 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065766 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0aabe90-1c40-4edb-8585-b5de80fdeeee-sys\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.065946 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065839 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cd70d5d-727b-46d9-aeb5-8b611c620946-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.065946 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0cd70d5d-727b-46d9-aeb5-8b611c620946-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.065946 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.065890 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-accelerators-collector-config\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.157737 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.157705 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:08.167006 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.166973 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.167175 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0aabe90-1c40-4edb-8585-b5de80fdeeee-sys\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.167175 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cd70d5d-727b-46d9-aeb5-8b611c620946-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.167175 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0cd70d5d-727b-46d9-aeb5-8b611c620946-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.167175 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-accelerators-collector-config\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.167175 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167143 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0aabe90-1c40-4edb-8585-b5de80fdeeee-metrics-client-ca\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.167726 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167183 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qztd\" (UniqueName: \"kubernetes.io/projected/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-api-access-5qztd\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.167726 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-tls\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.167726 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167272 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.167726 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167323 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-textfile\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.167726 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167220 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0aabe90-1c40-4edb-8585-b5de80fdeeee-sys\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.167726 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167664 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-textfile\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.167726 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167356 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c0aabe90-1c40-4edb-8585-b5de80fdeeee-root\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.168048 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167730 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-wtmp\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.168048 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167766 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.168048 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167791 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrt4\" (UniqueName: \"kubernetes.io/projected/c0aabe90-1c40-4edb-8585-b5de80fdeeee-kube-api-access-2zrt4\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.168048 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.168048 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.167920 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0aabe90-1c40-4edb-8585-b5de80fdeeee-metrics-client-ca\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.168274 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.168179 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.168274 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.168206 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cd70d5d-727b-46d9-aeb5-8b611c620946-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.168274 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.168244 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c0aabe90-1c40-4edb-8585-b5de80fdeeee-root\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.168418 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.168280 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0cd70d5d-727b-46d9-aeb5-8b611c620946-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.168418 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.168337 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-wtmp\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.168775 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.168744 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-accelerators-collector-config\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.173066 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.173040 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-tls\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.173066 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.173057 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0aabe90-1c40-4edb-8585-b5de80fdeeee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.173254 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.173172 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.173254 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.173173 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.181584 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.181554 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrt4\" (UniqueName: \"kubernetes.io/projected/c0aabe90-1c40-4edb-8585-b5de80fdeeee-kube-api-access-2zrt4\") pod \"node-exporter-kw5nb\" (UID: \"c0aabe90-1c40-4edb-8585-b5de80fdeeee\") " pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.181929 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.181907 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qztd\" (UniqueName: \"kubernetes.io/projected/0cd70d5d-727b-46d9-aeb5-8b611c620946-kube-api-access-5qztd\") pod \"kube-state-metrics-7c9dbb8b7d-24dwp\" (UID: \"0cd70d5d-727b-46d9-aeb5-8b611c620946\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.257750 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.257662 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:08.258508 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.258490 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" Jul 06 23:57:08.269183 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.269141 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:57:08.272646 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.272624 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:57:08.282637 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.282601 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1d07f90-5abf-453f-bc2a-920be5e548a8-original-pull-secret\") pod \"global-pull-secret-syncer-4qtnq\" (UID: \"f1d07f90-5abf-453f-bc2a-920be5e548a8\") " pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:57:08.317854 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.317819 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kw5nb" Jul 06 23:57:08.329032 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:57:08.328994 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0aabe90_1c40_4edb_8585_b5de80fdeeee.slice/crio-261e1d531610fdc437e1ead065fe076cdb27a94edfb98cbcee08658fddf4bd9f WatchSource:0}: Error finding container 261e1d531610fdc437e1ead065fe076cdb27a94edfb98cbcee08658fddf4bd9f: Status 404 returned error can't find the container with id 261e1d531610fdc437e1ead065fe076cdb27a94edfb98cbcee08658fddf4bd9f Jul 06 23:57:08.356053 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.355978 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:08.370120 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.370078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:57:08.370295 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.370172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:57:08.373497 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.373468 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:57:08.373642 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.373593 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:57:08.383272 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.383245 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305b651b-c1bc-4b0b-99fb-091bebcebb06-metrics-certs\") pod \"network-metrics-daemon-vsq4t\" (UID: \"305b651b-c1bc-4b0b-99fb-091bebcebb06\") " pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:57:08.383396 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.383361 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:57:08.391140 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.391112 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp"] Jul 06 23:57:08.393426 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.393402 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm59l\" (UniqueName: \"kubernetes.io/projected/612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc-kube-api-access-xm59l\") pod \"network-check-target-mkjcg\" (UID: \"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc\") " pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:57:08.394542 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:57:08.394510 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd70d5d_727b_46d9_aeb5_8b611c620946.slice/crio-ed5eb55fd89c888486decfd815614a501a5450f6b81e44a7aa7b2c9a52d1bcbf WatchSource:0}: Error finding container ed5eb55fd89c888486decfd815614a501a5450f6b81e44a7aa7b2c9a52d1bcbf: Status 404 returned error can't find the container with id ed5eb55fd89c888486decfd815614a501a5450f6b81e44a7aa7b2c9a52d1bcbf Jul 06 23:57:08.484493 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.484444 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qtnq" Jul 06 23:57:08.598172 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.597818 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dplr9\"" Jul 06 23:57:08.599300 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.599284 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-czj4k\"" Jul 06 23:57:08.605262 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.605238 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vsq4t" Jul 06 23:57:08.607498 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.607451 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:57:08.608914 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.608883 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4qtnq"] Jul 06 23:57:08.612941 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:57:08.612921 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d07f90_5abf_453f_bc2a_920be5e548a8.slice/crio-5f7a64971be7f5d86147d2d9587dd927db6645a03984853ca0763800315c6b51 WatchSource:0}: Error finding container 5f7a64971be7f5d86147d2d9587dd927db6645a03984853ca0763800315c6b51: Status 404 returned error can't find the container with id 5f7a64971be7f5d86147d2d9587dd927db6645a03984853ca0763800315c6b51 Jul 06 23:57:08.745073 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.745037 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mkjcg"] Jul 06 23:57:08.750081 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:57:08.750042 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612d21ce_89f6_4c6e_8d7d_0cdf5832c2fc.slice/crio-bc440fd68735c069a7cd8cd616b62c57c5e79dd1cc53110293c66b7c2ea04c70 WatchSource:0}: Error finding container bc440fd68735c069a7cd8cd616b62c57c5e79dd1cc53110293c66b7c2ea04c70: Status 404 returned error can't find the container with id bc440fd68735c069a7cd8cd616b62c57c5e79dd1cc53110293c66b7c2ea04c70 Jul 06 23:57:08.761360 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.761333 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vsq4t"] Jul 06 23:57:08.764102 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:57:08.764073 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305b651b_c1bc_4b0b_99fb_091bebcebb06.slice/crio-d6ed1dcb78321dd6e25049e1fde3312a50445725fe51d44d99819154bbbefa78 WatchSource:0}: Error finding container d6ed1dcb78321dd6e25049e1fde3312a50445725fe51d44d99819154bbbefa78: Status 404 returned error can't find the container with id d6ed1dcb78321dd6e25049e1fde3312a50445725fe51d44d99819154bbbefa78 Jul 06 23:57:08.840285 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.840188 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:08.923578 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.923509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:08.937748 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.937720 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kps6x" Jul 06 23:57:08.965878 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.965837 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mkjcg" event={"ID":"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc","Type":"ContainerStarted","Data":"bc440fd68735c069a7cd8cd616b62c57c5e79dd1cc53110293c66b7c2ea04c70"} Jul 06 23:57:08.967337 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.967303 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kw5nb" event={"ID":"c0aabe90-1c40-4edb-8585-b5de80fdeeee","Type":"ContainerStarted","Data":"261e1d531610fdc437e1ead065fe076cdb27a94edfb98cbcee08658fddf4bd9f"} Jul 06 23:57:08.976957 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.970591 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vsq4t" event={"ID":"305b651b-c1bc-4b0b-99fb-091bebcebb06","Type":"ContainerStarted","Data":"d6ed1dcb78321dd6e25049e1fde3312a50445725fe51d44d99819154bbbefa78"} Jul 06 23:57:08.976957 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.972336 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4qtnq" event={"ID":"f1d07f90-5abf-453f-bc2a-920be5e548a8","Type":"ContainerStarted","Data":"5f7a64971be7f5d86147d2d9587dd927db6645a03984853ca0763800315c6b51"} Jul 06 23:57:08.976957 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:08.974427 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" event={"ID":"0cd70d5d-727b-46d9-aeb5-8b611c620946","Type":"ContainerStarted","Data":"ed5eb55fd89c888486decfd815614a501a5450f6b81e44a7aa7b2c9a52d1bcbf"} Jul 06 23:57:09.361282 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.361245 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:09.920893 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.920851 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5bc589d866-pgm7r"] Jul 06 23:57:09.929365 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.925636 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:09.929365 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.928623 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 23:57:09.929365 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.928824 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 23:57:09.929365 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.928914 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 23:57:09.929365 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.929071 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-klp9p\"" Jul 06 23:57:09.929365 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.929284 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 23:57:09.929365 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.929308 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 23:57:09.929829 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.929724 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2ouj0o4c2abam\"" Jul 06 23:57:09.939097 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.938998 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bc589d866-pgm7r"] Jul 06 23:57:09.984801 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.984763 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23cbc791-6397-465e-b1a9-f294ca405fc1-metrics-client-ca\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:09.984986 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.984827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:09.984986 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.984859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:09.984986 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.984924 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqn6\" (UniqueName: \"kubernetes.io/projected/23cbc791-6397-465e-b1a9-f294ca405fc1-kube-api-access-zkqn6\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:09.984986 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.984968 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:09.985235 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.984993 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:09.985235 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.985037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-tls\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:09.985235 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:09.985064 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-grpc-tls\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.086979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.086102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqn6\" (UniqueName: \"kubernetes.io/projected/23cbc791-6397-465e-b1a9-f294ca405fc1-kube-api-access-zkqn6\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.086979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.086142 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.086979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.086176 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.086979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.086217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-tls\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.086979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.086249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-grpc-tls\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.086979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.086278 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23cbc791-6397-465e-b1a9-f294ca405fc1-metrics-client-ca\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.086979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.086380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.086979 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.086413 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.087678 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.087368 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23cbc791-6397-465e-b1a9-f294ca405fc1-metrics-client-ca\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.090577 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.090134 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-tls\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.090577 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.090514 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.092294 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.092267 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.092412 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.092267 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.092656 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.092630 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-grpc-tls\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.092733 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.092718 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23cbc791-6397-465e-b1a9-f294ca405fc1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.096783 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.096733 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqn6\" (UniqueName: \"kubernetes.io/projected/23cbc791-6397-465e-b1a9-f294ca405fc1-kube-api-access-zkqn6\") pod \"thanos-querier-5bc589d866-pgm7r\" (UID: \"23cbc791-6397-465e-b1a9-f294ca405fc1\") " pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.134425 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.134333 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:10.234418 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.234257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:10.244262 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.244228 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:10.524488 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.524451 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bc589d866-pgm7r"] Jul 06 23:57:10.526885 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:57:10.526854 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cbc791_6397_465e_b1a9_f294ca405fc1.slice/crio-2278bf8638ff928ab587650702f8657992ac6e1d52d9122773712593d4498ac1 WatchSource:0}: Error finding container 2278bf8638ff928ab587650702f8657992ac6e1d52d9122773712593d4498ac1: Status 404 returned error can't find the container with id 2278bf8638ff928ab587650702f8657992ac6e1d52d9122773712593d4498ac1 Jul 06 23:57:10.845592 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.845483 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:10.943157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.943119 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:10.987554 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.987206 2563 generic.go:356] "Generic (PLEG): container finished" podID="c0aabe90-1c40-4edb-8585-b5de80fdeeee" containerID="02f1c78eec6ea5c7338dc556f6c2d1c1041363b7aaf71d17e272c123d989fdfc" exitCode=0 Jul 06 23:57:10.987554 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.987301 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kw5nb" event={"ID":"c0aabe90-1c40-4edb-8585-b5de80fdeeee","Type":"ContainerDied","Data":"02f1c78eec6ea5c7338dc556f6c2d1c1041363b7aaf71d17e272c123d989fdfc"} Jul 06 23:57:10.989555 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.989498 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" event={"ID":"23cbc791-6397-465e-b1a9-f294ca405fc1","Type":"ContainerStarted","Data":"2278bf8638ff928ab587650702f8657992ac6e1d52d9122773712593d4498ac1"} Jul 06 23:57:10.992687 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.992661 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vsq4t" event={"ID":"305b651b-c1bc-4b0b-99fb-091bebcebb06","Type":"ContainerStarted","Data":"87797bfab213c67b84e8f4958fb1b03a02cb3d042dd6606a54e99e252725fb83"} Jul 06 23:57:10.992818 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.992731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vsq4t" event={"ID":"305b651b-c1bc-4b0b-99fb-091bebcebb06","Type":"ContainerStarted","Data":"d09c3acaded80404f6ab042366b7f92b167b6621917a6a0a0f60a11cb79a664e"} Jul 06 23:57:10.995879 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.995844 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" event={"ID":"0cd70d5d-727b-46d9-aeb5-8b611c620946","Type":"ContainerStarted","Data":"5d81d3aa6a086c2360605bea487365e8e1e5939a8a882c06b4bd9d213f80702a"} Jul 06 23:57:10.995985 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.995903 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" event={"ID":"0cd70d5d-727b-46d9-aeb5-8b611c620946","Type":"ContainerStarted","Data":"ff928f6dbfcca2f298a95f3dbc81084e440f6e6124731bad5c59679382a4f189"} Jul 06 23:57:10.995985 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:10.995924 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" event={"ID":"0cd70d5d-727b-46d9-aeb5-8b611c620946","Type":"ContainerStarted","Data":"7a5e8c79789b65ee1ce771a29cc59ffff6e71920521b020329d668da00bca4dc"} Jul 06 23:57:11.027560 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:11.027473 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vsq4t" podStartSLOduration=66.467174568 podStartE2EDuration="1m8.027448747s" totalImagesPullingTime="1.560274179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:11.027070156 +0000 UTC m=+68.972655528" watchObservedRunningTime="2026-07-06 23:57:11.027448747 +0000 UTC m=+68.973034119" Jul 06 23:57:11.046813 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:11.046747 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-24dwp" podStartSLOduration=2.115423428 podStartE2EDuration="4.046723541s" totalImagesPullingTime="1.931300113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:11.045924943 +0000 UTC m=+68.991510314" watchObservedRunningTime="2026-07-06 23:57:11.046723541 +0000 UTC m=+68.992308919" Jul 06 23:57:11.130824 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:11.130765 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:11.196568 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:11.196475 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:11.440028 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:11.439957 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:11.540866 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:11.540808 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:12.003728 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.003677 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kw5nb" event={"ID":"c0aabe90-1c40-4edb-8585-b5de80fdeeee","Type":"ContainerStarted","Data":"74e82edd66d7e57b977557050636a47c8375e57335efd5976133b06f118280cb"} Jul 06 23:57:12.003728 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.003746 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kw5nb" event={"ID":"c0aabe90-1c40-4edb-8585-b5de80fdeeee","Type":"ContainerStarted","Data":"706132bdc9410efa4b35004b98578743433787e89a57a4c923dbe8b9039d6c91"} Jul 06 23:57:12.028018 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.027935 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kw5nb" podStartSLOduration=2.905443702 podStartE2EDuration="5.027912869s" totalImagesPullingTime="1.995035236s" totalInitContainerRuntime="127.433931ms" isStatefulPod=true podCreationTimestamp="2026-07-06 23:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:12.026678945 +0000 UTC m=+69.972264339" watchObservedRunningTime="2026-07-06 23:57:12.027912869 +0000 UTC m=+69.973498241" Jul 06 23:57:12.238631 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.238588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:12.266439 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.266351 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-767fc4b76f-vjxx9"] Jul 06 23:57:12.286157 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.286122 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.287687 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.287660 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-767fc4b76f-vjxx9"] Jul 06 23:57:12.289607 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.289565 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 23:57:12.291232 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.290966 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-689wq\"" Jul 06 23:57:12.291232 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.291006 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 23:57:12.291232 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.291037 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bfhjlgdr6unp7\"" Jul 06 23:57:12.291554 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.291266 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 23:57:12.291554 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.291516 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 23:57:12.313108 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.313064 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:12.383277 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.383242 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:12.408357 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.408313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/368845f5-c1ee-42d0-a210-b083f94eaf04-audit-log\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.408569 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.408422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/368845f5-c1ee-42d0-a210-b083f94eaf04-secret-metrics-server-tls\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.408569 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.408468 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6l7r\" (UniqueName: \"kubernetes.io/projected/368845f5-c1ee-42d0-a210-b083f94eaf04-kube-api-access-j6l7r\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.408685 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.408576 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368845f5-c1ee-42d0-a210-b083f94eaf04-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.408685 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.408651 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/368845f5-c1ee-42d0-a210-b083f94eaf04-secret-metrics-server-client-certs\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.408774 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.408687 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/368845f5-c1ee-42d0-a210-b083f94eaf04-client-ca-bundle\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.408774 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.408714 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/368845f5-c1ee-42d0-a210-b083f94eaf04-metrics-server-audit-profiles\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.509906 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.509871 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/368845f5-c1ee-42d0-a210-b083f94eaf04-client-ca-bundle\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.510106 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.509919 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/368845f5-c1ee-42d0-a210-b083f94eaf04-metrics-server-audit-profiles\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.510106 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.509954 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/368845f5-c1ee-42d0-a210-b083f94eaf04-audit-log\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.510106 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.509986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/368845f5-c1ee-42d0-a210-b083f94eaf04-secret-metrics-server-tls\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.510106 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.510013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6l7r\" (UniqueName: \"kubernetes.io/projected/368845f5-c1ee-42d0-a210-b083f94eaf04-kube-api-access-j6l7r\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.510106 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.510049 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368845f5-c1ee-42d0-a210-b083f94eaf04-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.510106 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.510089 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/368845f5-c1ee-42d0-a210-b083f94eaf04-secret-metrics-server-client-certs\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.511078 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.511036 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368845f5-c1ee-42d0-a210-b083f94eaf04-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.511337 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.511312 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/368845f5-c1ee-42d0-a210-b083f94eaf04-audit-log\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.511576 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.511523 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/368845f5-c1ee-42d0-a210-b083f94eaf04-metrics-server-audit-profiles\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.512768 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.512746 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/368845f5-c1ee-42d0-a210-b083f94eaf04-client-ca-bundle\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.512869 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.512853 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/368845f5-c1ee-42d0-a210-b083f94eaf04-secret-metrics-server-client-certs\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.512936 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.512871 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/368845f5-c1ee-42d0-a210-b083f94eaf04-secret-metrics-server-tls\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.520274 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.520210 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6l7r\" (UniqueName: \"kubernetes.io/projected/368845f5-c1ee-42d0-a210-b083f94eaf04-kube-api-access-j6l7r\") pod \"metrics-server-767fc4b76f-vjxx9\" (UID: \"368845f5-c1ee-42d0-a210-b083f94eaf04\") " pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.599925 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.599888 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:12.734455 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.734415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:12.878280 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:12.878183 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:13.026736 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:13.026685 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:13.253285 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:13.253244 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:13.358653 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:13.358615 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:13.915365 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:13.915337 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84b64754fb-tdx8n" Jul 06 23:57:14.122885 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:14.122837 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:14.255888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:14.255805 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:14.328643 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:14.328605 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:14.947583 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:14.947553 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-767fc4b76f-vjxx9"] Jul 06 23:57:14.951152 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:57:14.951116 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368845f5_c1ee_42d0_a210_b083f94eaf04.slice/crio-e5d78e13f9e281400ea64a7ecd02fff409fb2d873b483f8c571ac27db54e4b54 WatchSource:0}: Error finding container e5d78e13f9e281400ea64a7ecd02fff409fb2d873b483f8c571ac27db54e4b54: Status 404 returned error can't find the container with id e5d78e13f9e281400ea64a7ecd02fff409fb2d873b483f8c571ac27db54e4b54 Jul 06 23:57:15.018027 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.017960 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4qtnq" event={"ID":"f1d07f90-5abf-453f-bc2a-920be5e548a8","Type":"ContainerStarted","Data":"11139b8752daf910050fc7c494e7521041b94cf8ab2121e84e32f1a40d8fa131"} Jul 06 23:57:15.019787 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.019758 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" event={"ID":"368845f5-c1ee-42d0-a210-b083f94eaf04","Type":"ContainerStarted","Data":"e5d78e13f9e281400ea64a7ecd02fff409fb2d873b483f8c571ac27db54e4b54"} Jul 06 23:57:15.021434 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.021411 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mkjcg" event={"ID":"612d21ce-89f6-4c6e-8d7d-0cdf5832c2fc","Type":"ContainerStarted","Data":"a9dde180626fcb6e20edd889b078fd7c2402812292dc7ecc86a348e4cb39a16a"} Jul 06 23:57:15.021778 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.021709 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:57:15.023809 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.023787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" event={"ID":"23cbc791-6397-465e-b1a9-f294ca405fc1","Type":"ContainerStarted","Data":"cee19e214940ba90a3212d7c5eb8040de1f09ff554197035af71e5479bf10f21"} Jul 06 23:57:15.023908 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.023839 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" event={"ID":"23cbc791-6397-465e-b1a9-f294ca405fc1","Type":"ContainerStarted","Data":"c45ab891da44d085b7f1d5ddd7da5a5040da7dbfdab269a4411fc938e8dc9214"} Jul 06 23:57:15.035726 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.035671 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4qtnq" podStartSLOduration=65.841257791 podStartE2EDuration="1m12.035647832s" totalImagesPullingTime="6.194390041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:15.034915918 +0000 UTC m=+72.980501290" watchObservedRunningTime="2026-07-06 23:57:15.035647832 +0000 UTC m=+72.981233201" Jul 06 23:57:15.054703 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.054645 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mkjcg" podStartSLOduration=66.011632301 podStartE2EDuration="1m12.054625352s" totalImagesPullingTime="6.042993051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:15.053127009 +0000 UTC m=+72.998712381" watchObservedRunningTime="2026-07-06 23:57:15.054625352 +0000 UTC m=+73.000210723" Jul 06 23:57:15.126827 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.126789 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:15.196090 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:15.196055 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:16.031091 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:16.031050 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" event={"ID":"23cbc791-6397-465e-b1a9-f294ca405fc1","Type":"ContainerStarted","Data":"6e8f5a5e3b16e354607bfccca8631fa2fa56b30274ae06d5a5001e20cb0ceee2"} Jul 06 23:57:17.042480 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.042441 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" event={"ID":"23cbc791-6397-465e-b1a9-f294ca405fc1","Type":"ContainerStarted","Data":"2adf0d616d4feb4bedb18b276b38cb164afbb3dfc4ca746920020339d30ec280"} Jul 06 23:57:17.042993 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.042584 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" event={"ID":"23cbc791-6397-465e-b1a9-f294ca405fc1","Type":"ContainerStarted","Data":"c97f88c38f3a60abba29a545e485f47c8bf9c6dd7ad7bd7849af0c67eab9f816"} Jul 06 23:57:17.042993 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.042633 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" event={"ID":"23cbc791-6397-465e-b1a9-f294ca405fc1","Type":"ContainerStarted","Data":"b23a99c2bb075650693ca8606587a0d22357fd6f29d0040f7222cd3d0f1f2f8f"} Jul 06 23:57:17.042993 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.042680 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:17.044040 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.044020 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" event={"ID":"368845f5-c1ee-42d0-a210-b083f94eaf04","Type":"ContainerStarted","Data":"087b6dbed1200cb808be799277a81a9059d3ecaedec4cdbd8155c1dd23901384"} Jul 06 23:57:17.075399 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.075343 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" podStartSLOduration=2.435997662 podStartE2EDuration="8.075324235s" totalImagesPullingTime="5.639326573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:17.074706509 +0000 UTC m=+75.020291877" watchObservedRunningTime="2026-07-06 23:57:17.075324235 +0000 UTC m=+75.020909606" Jul 06 23:57:17.101819 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.101756 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" podStartSLOduration=3.641624086 podStartE2EDuration="5.101731747s" totalImagesPullingTime="1.460107661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:17.100160161 +0000 UTC m=+75.045745531" watchObservedRunningTime="2026-07-06 23:57:17.101731747 +0000 UTC m=+75.047317118" Jul 06 23:57:17.446781 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.446739 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:17.611636 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.611603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:17.715283 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:17.715173 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:19.406850 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:19.406805 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:19.501849 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:19.501807 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:20.922219 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:20.922182 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:21.090024 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:21.089991 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:21.547684 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:21.547651 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:21.655942 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:21.655908 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:23.056238 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:23.056107 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5bc589d866-pgm7r" Jul 06 23:57:24.153522 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:24.153489 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:24.331118 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:24.331085 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:27.306341 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:27.306295 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:27.366093 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:27.366062 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:27.995377 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:27.995343 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:28.098366 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:28.098335 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:28.196113 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:28.196087 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:28.301133 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:28.301041 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:32.601029 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:32.600987 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:32.601500 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:32.601057 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:42.313471 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:42.313439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:42.371491 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:42.371457 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:46.034196 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:46.034159 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mkjcg" Jul 06 23:57:52.605927 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:52.605895 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:52.610074 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:52.610049 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-767fc4b76f-vjxx9" Jul 06 23:57:57.254855 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:57.254819 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:57:57.567649 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:57:57.567559 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:58:02.366661 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.366627 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt"] Jul 06 23:58:02.372160 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.372127 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.374819 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.374791 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 23:58:02.375424 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.375395 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 23:58:02.375632 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.375616 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8xfqq\"" Jul 06 23:58:02.375779 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.375669 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 23:58:02.375906 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.375876 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 23:58:02.375992 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.375916 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 23:58:02.382020 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.382001 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 23:58:02.389897 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.389875 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt"] Jul 06 23:58:02.406381 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.406347 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.406603 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.406400 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-serving-certs-ca-bundle\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.406603 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.406430 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-federate-client-tls\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.406603 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.406457 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.406603 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.406508 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-secret-telemeter-client\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.406791 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.406604 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-metrics-client-ca\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.406791 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.406632 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcx7p\" (UniqueName: \"kubernetes.io/projected/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-kube-api-access-rcx7p\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.406791 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.406659 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-telemeter-client-tls\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.507826 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.507786 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-federate-client-tls\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.507998 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.507844 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.507998 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.507885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-secret-telemeter-client\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.507998 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.507925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-metrics-client-ca\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.507998 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.507948 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcx7p\" (UniqueName: \"kubernetes.io/projected/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-kube-api-access-rcx7p\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.507998 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.507970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-telemeter-client-tls\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.508456 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.508028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.508456 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.508071 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-serving-certs-ca-bundle\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.508904 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.508881 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-metrics-client-ca\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.510726 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.510697 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 23:58:02.510830 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.510744 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 23:58:02.511409 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.511395 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 23:58:02.511494 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.511412 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 23:58:02.511774 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.511757 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 23:58:02.518059 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.518028 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 23:58:02.519193 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.519160 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-serving-certs-ca-bundle\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.519985 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.519382 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.520138 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.519919 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcx7p\" (UniqueName: \"kubernetes.io/projected/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-kube-api-access-rcx7p\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.521037 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.521014 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-secret-telemeter-client\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.521153 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.521131 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-federate-client-tls\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.521286 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.521270 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-telemeter-client-tls\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.521327 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.521267 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c6bc74d8b-2kxkt\" (UID: \"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0\") " pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.683663 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.683635 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8xfqq\"" Jul 06 23:58:02.693864 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.693817 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" Jul 06 23:58:02.843850 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:02.843812 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt"] Jul 06 23:58:02.846934 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:58:02.846907 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c3d8d9_9b19_49e5_9d52_7c13465a5ed0.slice/crio-4eb5a0a152d86374a347814147d06c7109252c8ba96246023ef8fb6cf6e8c07e WatchSource:0}: Error finding container 4eb5a0a152d86374a347814147d06c7109252c8ba96246023ef8fb6cf6e8c07e: Status 404 returned error can't find the container with id 4eb5a0a152d86374a347814147d06c7109252c8ba96246023ef8fb6cf6e8c07e Jul 06 23:58:03.208600 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:03.208565 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" event={"ID":"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0","Type":"ContainerStarted","Data":"4eb5a0a152d86374a347814147d06c7109252c8ba96246023ef8fb6cf6e8c07e"} Jul 06 23:58:05.218746 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:05.218711 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" event={"ID":"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0","Type":"ContainerStarted","Data":"d8eefaa4120236e24c36578aac8a2b4130fff489a59290bb57143cc37a1bb3d3"} Jul 06 23:58:06.224421 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:06.224379 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" event={"ID":"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0","Type":"ContainerStarted","Data":"01b0f78ccbeea05c0e816477c2f1d14163189533c1cf3a865659ada97d11262a"} Jul 06 23:58:06.224842 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:06.224447 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" event={"ID":"c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0","Type":"ContainerStarted","Data":"6671324c1042b10c56ab03b0d09f6084e69e6d82387d93cf4aa764ee819b7abb"} Jul 06 23:58:06.246771 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:06.246714 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5c6bc74d8b-2kxkt" podStartSLOduration=1.845054934 podStartE2EDuration="4.246687166s" totalImagesPullingTime="2.401632232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:58:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:06.24530507 +0000 UTC m=+124.190890454" watchObservedRunningTime="2026-07-06 23:58:06.246687166 +0000 UTC m=+124.192272537" Jul 06 23:58:07.007300 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:07.007263 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:58:07.108224 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:07.108185 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:58:18.303798 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:18.303753 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:58:18.404518 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:18.404483 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:58:18.461806 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:18.461767 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:58:38.431012 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:38.430979 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:58:38.496802 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:38.496768 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:58:39.416672 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:58:39.416640 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:12.505216 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:12.505183 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:12.564729 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:12.564690 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:16.394884 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:16.394849 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:16.528888 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:16.528858 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:17.217961 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:17.217889 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:17.327299 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:17.327250 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:17.893487 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:17.893441 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:24.485044 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:24.485007 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:24.563836 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:24.563797 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:27.302443 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:27.302412 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:27.400865 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:27.400830 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:27.502141 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:27.502083 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:27.605150 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:27.605060 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:27.667214 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:27.667178 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:27.724249 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:27.724187 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:50.721488 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:50.721454 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:50.815891 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:50.815859 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 06 23:59:51.343689 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.343654 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft"] Jul 06 23:59:51.346957 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.346937 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.349697 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.349674 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 23:59:51.349838 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.349715 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jrqh2\"" Jul 06 23:59:51.349838 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.349715 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 23:59:51.354816 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.354783 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft"] Jul 06 23:59:51.461798 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.461757 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zgf7\" (UniqueName: \"kubernetes.io/projected/39c5ce8c-76e8-4c41-8550-05a7164f6a47-kube-api-access-5zgf7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.461978 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.461809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.461978 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.461834 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.562698 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.562660 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zgf7\" (UniqueName: \"kubernetes.io/projected/39c5ce8c-76e8-4c41-8550-05a7164f6a47-kube-api-access-5zgf7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.562870 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.562710 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.562870 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.562739 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.563118 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.563098 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.563188 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.563162 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.571102 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.571069 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zgf7\" (UniqueName: \"kubernetes.io/projected/39c5ce8c-76e8-4c41-8550-05a7164f6a47-kube-api-access-5zgf7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.657495 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.657464 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 06 23:59:51.789138 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:51.789106 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft"] Jul 06 23:59:51.792152 ip-10-0-133-6 kubenswrapper[2563]: W0706 23:59:51.792126 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c5ce8c_76e8_4c41_8550_05a7164f6a47.slice/crio-92c95b2826bef548a8c7b0cbd52fc12abd71bf592be4af30d1f99b280dda51e7 WatchSource:0}: Error finding container 92c95b2826bef548a8c7b0cbd52fc12abd71bf592be4af30d1f99b280dda51e7: Status 404 returned error can't find the container with id 92c95b2826bef548a8c7b0cbd52fc12abd71bf592be4af30d1f99b280dda51e7 Jul 06 23:59:52.600249 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:52.600204 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" event={"ID":"39c5ce8c-76e8-4c41-8550-05a7164f6a47","Type":"ContainerStarted","Data":"92c95b2826bef548a8c7b0cbd52fc12abd71bf592be4af30d1f99b280dda51e7"} Jul 06 23:59:57.620963 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:57.620919 2563 generic.go:356] "Generic (PLEG): container finished" podID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerID="08664f81d4cafef93eeaea2b42f33ff02cdb4c8e7db0902a80e289b62e890b85" exitCode=0 Jul 06 23:59:57.621376 ip-10-0-133-6 kubenswrapper[2563]: I0706 23:59:57.620992 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" event={"ID":"39c5ce8c-76e8-4c41-8550-05a7164f6a47","Type":"ContainerDied","Data":"08664f81d4cafef93eeaea2b42f33ff02cdb4c8e7db0902a80e289b62e890b85"} Jul 07 00:00:00.182386 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.182355 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29723040-z8bg8"] Jul 07 00:00:00.185323 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.185306 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:00.188604 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.188577 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-tpl5x\"" Jul 07 00:00:00.188754 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.188714 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Jul 07 00:00:00.209977 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.209784 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29723040-z8bg8"] Jul 07 00:00:00.338424 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.338384 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38b8c49-6cc0-4234-8b07-4b957681ca29-serviceca\") pod \"image-pruner-29723040-z8bg8\" (UID: \"e38b8c49-6cc0-4234-8b07-4b957681ca29\") " pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:00.338627 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.338437 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxf8\" (UniqueName: \"kubernetes.io/projected/e38b8c49-6cc0-4234-8b07-4b957681ca29-kube-api-access-4wxf8\") pod \"image-pruner-29723040-z8bg8\" (UID: \"e38b8c49-6cc0-4234-8b07-4b957681ca29\") " pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:00.439481 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.439404 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38b8c49-6cc0-4234-8b07-4b957681ca29-serviceca\") pod \"image-pruner-29723040-z8bg8\" (UID: \"e38b8c49-6cc0-4234-8b07-4b957681ca29\") " pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:00.439481 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.439456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxf8\" (UniqueName: \"kubernetes.io/projected/e38b8c49-6cc0-4234-8b07-4b957681ca29-kube-api-access-4wxf8\") pod \"image-pruner-29723040-z8bg8\" (UID: \"e38b8c49-6cc0-4234-8b07-4b957681ca29\") " pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:00.440001 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.439982 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38b8c49-6cc0-4234-8b07-4b957681ca29-serviceca\") pod \"image-pruner-29723040-z8bg8\" (UID: \"e38b8c49-6cc0-4234-8b07-4b957681ca29\") " pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:00.450136 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.450111 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxf8\" (UniqueName: \"kubernetes.io/projected/e38b8c49-6cc0-4234-8b07-4b957681ca29-kube-api-access-4wxf8\") pod \"image-pruner-29723040-z8bg8\" (UID: \"e38b8c49-6cc0-4234-8b07-4b957681ca29\") " pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:00.519002 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.518964 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:00.634731 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.634695 2563 generic.go:356] "Generic (PLEG): container finished" podID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerID="7c327054d65c6a901a6b4749bcbbdb55cef7f1b7f69c00f9f6f9cde938fb9845" exitCode=0 Jul 07 00:00:00.634914 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.634778 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" event={"ID":"39c5ce8c-76e8-4c41-8550-05a7164f6a47","Type":"ContainerDied","Data":"7c327054d65c6a901a6b4749bcbbdb55cef7f1b7f69c00f9f6f9cde938fb9845"} Jul 07 00:00:00.658696 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:00.658605 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29723040-z8bg8"] Jul 07 00:00:00.661216 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:00:00.661191 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38b8c49_6cc0_4234_8b07_4b957681ca29.slice/crio-8dadf4ccccd48d506f87d2f57134ce93fea01531633cd05bf2d58ab3448227a6 WatchSource:0}: Error finding container 8dadf4ccccd48d506f87d2f57134ce93fea01531633cd05bf2d58ab3448227a6: Status 404 returned error can't find the container with id 8dadf4ccccd48d506f87d2f57134ce93fea01531633cd05bf2d58ab3448227a6 Jul 07 00:00:01.639957 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:01.639923 2563 generic.go:356] "Generic (PLEG): container finished" podID="e38b8c49-6cc0-4234-8b07-4b957681ca29" containerID="a26d24ebe257870f7f6f00959f57e3dd367429d3a0b1c31900b78011b2a2b478" exitCode=0 Jul 07 00:00:01.640327 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:01.639992 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29723040-z8bg8" event={"ID":"e38b8c49-6cc0-4234-8b07-4b957681ca29","Type":"ContainerDied","Data":"a26d24ebe257870f7f6f00959f57e3dd367429d3a0b1c31900b78011b2a2b478"} Jul 07 00:00:01.640327 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:01.640026 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29723040-z8bg8" event={"ID":"e38b8c49-6cc0-4234-8b07-4b957681ca29","Type":"ContainerStarted","Data":"8dadf4ccccd48d506f87d2f57134ce93fea01531633cd05bf2d58ab3448227a6"} Jul 07 00:00:02.793850 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:02.793827 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:02.861127 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:02.861089 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wxf8\" (UniqueName: \"kubernetes.io/projected/e38b8c49-6cc0-4234-8b07-4b957681ca29-kube-api-access-4wxf8\") pod \"e38b8c49-6cc0-4234-8b07-4b957681ca29\" (UID: \"e38b8c49-6cc0-4234-8b07-4b957681ca29\") " Jul 07 00:00:02.861293 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:02.861151 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38b8c49-6cc0-4234-8b07-4b957681ca29-serviceca\") pod \"e38b8c49-6cc0-4234-8b07-4b957681ca29\" (UID: \"e38b8c49-6cc0-4234-8b07-4b957681ca29\") " Jul 07 00:00:02.861591 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:02.861564 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38b8c49-6cc0-4234-8b07-4b957681ca29-serviceca" (OuterVolumeSpecName: "serviceca") pod "e38b8c49-6cc0-4234-8b07-4b957681ca29" (UID: "e38b8c49-6cc0-4234-8b07-4b957681ca29"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:00:02.863743 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:02.863716 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38b8c49-6cc0-4234-8b07-4b957681ca29-kube-api-access-4wxf8" (OuterVolumeSpecName: "kube-api-access-4wxf8") pod "e38b8c49-6cc0-4234-8b07-4b957681ca29" (UID: "e38b8c49-6cc0-4234-8b07-4b957681ca29"). InnerVolumeSpecName "kube-api-access-4wxf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:00:02.962336 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:02.962255 2563 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38b8c49-6cc0-4234-8b07-4b957681ca29-serviceca\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:02.962336 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:02.962286 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxf8\" (UniqueName: \"kubernetes.io/projected/e38b8c49-6cc0-4234-8b07-4b957681ca29-kube-api-access-4wxf8\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:03.650210 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:03.650179 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29723040-z8bg8" Jul 07 00:00:03.650210 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:03.650187 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29723040-z8bg8" event={"ID":"e38b8c49-6cc0-4234-8b07-4b957681ca29","Type":"ContainerDied","Data":"8dadf4ccccd48d506f87d2f57134ce93fea01531633cd05bf2d58ab3448227a6"} Jul 07 00:00:03.650430 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:03.650242 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dadf4ccccd48d506f87d2f57134ce93fea01531633cd05bf2d58ab3448227a6" Jul 07 00:00:08.671844 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:08.671809 2563 generic.go:356] "Generic (PLEG): container finished" podID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerID="5289048a71c6e0ca72d0b4c04f03e47456f933a55bf8c9b1cfe8ab4170e5c470" exitCode=0 Jul 07 00:00:08.672221 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:08.671856 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" event={"ID":"39c5ce8c-76e8-4c41-8550-05a7164f6a47","Type":"ContainerDied","Data":"5289048a71c6e0ca72d0b4c04f03e47456f933a55bf8c9b1cfe8ab4170e5c470"} Jul 07 00:00:09.801579 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.801554 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 07 00:00:09.814704 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.814679 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zgf7\" (UniqueName: \"kubernetes.io/projected/39c5ce8c-76e8-4c41-8550-05a7164f6a47-kube-api-access-5zgf7\") pod \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " Jul 07 00:00:09.814857 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.814761 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-util\") pod \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " Jul 07 00:00:09.814857 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.814792 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-bundle\") pod \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\" (UID: \"39c5ce8c-76e8-4c41-8550-05a7164f6a47\") " Jul 07 00:00:09.815396 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.815368 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-bundle" (OuterVolumeSpecName: "bundle") pod "39c5ce8c-76e8-4c41-8550-05a7164f6a47" (UID: "39c5ce8c-76e8-4c41-8550-05a7164f6a47"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:00:09.816989 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.816966 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c5ce8c-76e8-4c41-8550-05a7164f6a47-kube-api-access-5zgf7" (OuterVolumeSpecName: "kube-api-access-5zgf7") pod "39c5ce8c-76e8-4c41-8550-05a7164f6a47" (UID: "39c5ce8c-76e8-4c41-8550-05a7164f6a47"). InnerVolumeSpecName "kube-api-access-5zgf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:00:09.819167 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.819143 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-util" (OuterVolumeSpecName: "util") pod "39c5ce8c-76e8-4c41-8550-05a7164f6a47" (UID: "39c5ce8c-76e8-4c41-8550-05a7164f6a47"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:00:09.915898 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.915846 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-util\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:09.915898 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.915890 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5ce8c-76e8-4c41-8550-05a7164f6a47-bundle\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:09.915898 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:09.915899 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zgf7\" (UniqueName: \"kubernetes.io/projected/39c5ce8c-76e8-4c41-8550-05a7164f6a47-kube-api-access-5zgf7\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:10.682620 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:10.682581 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" event={"ID":"39c5ce8c-76e8-4c41-8550-05a7164f6a47","Type":"ContainerDied","Data":"92c95b2826bef548a8c7b0cbd52fc12abd71bf592be4af30d1f99b280dda51e7"} Jul 07 00:00:10.682781 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:10.682629 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xhkft" Jul 07 00:00:10.682781 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:10.682634 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92c95b2826bef548a8c7b0cbd52fc12abd71bf592be4af30d1f99b280dda51e7" Jul 07 00:00:12.389801 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:12.389756 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:13.220757 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:13.220726 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:14.326811 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.326775 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km"] Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327027 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerName="extract" Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327037 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerName="extract" Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327051 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e38b8c49-6cc0-4234-8b07-4b957681ca29" containerName="image-pruner" Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327056 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e38b8c49-6cc0-4234-8b07-4b957681ca29" containerName="image-pruner" Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327069 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerName="pull" Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327075 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerName="pull" Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327081 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerName="util" Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327087 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerName="util" Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327136 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e38b8c49-6cc0-4234-8b07-4b957681ca29" containerName="image-pruner" Jul 07 00:00:14.327238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.327144 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c5ce8c-76e8-4c41-8550-05a7164f6a47" containerName="extract" Jul 07 00:00:14.367607 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.367568 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km"] Jul 07 00:00:14.367791 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.367622 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" Jul 07 00:00:14.371527 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.371327 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 07 00:00:14.371527 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.371364 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 07 00:00:14.371752 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.371718 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pn8zk\"" Jul 07 00:00:14.448063 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.448025 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12d4685a-c83b-4010-8a94-7072edd46188-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9m2km\" (UID: \"12d4685a-c83b-4010-8a94-7072edd46188\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" Jul 07 00:00:14.448230 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.448085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nncmj\" (UniqueName: \"kubernetes.io/projected/12d4685a-c83b-4010-8a94-7072edd46188-kube-api-access-nncmj\") pod \"cert-manager-operator-controller-manager-55c444bf86-9m2km\" (UID: \"12d4685a-c83b-4010-8a94-7072edd46188\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" Jul 07 00:00:14.548975 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.548942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12d4685a-c83b-4010-8a94-7072edd46188-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9m2km\" (UID: \"12d4685a-c83b-4010-8a94-7072edd46188\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" Jul 07 00:00:14.549157 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.548997 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nncmj\" (UniqueName: \"kubernetes.io/projected/12d4685a-c83b-4010-8a94-7072edd46188-kube-api-access-nncmj\") pod \"cert-manager-operator-controller-manager-55c444bf86-9m2km\" (UID: \"12d4685a-c83b-4010-8a94-7072edd46188\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" Jul 07 00:00:14.549374 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.549351 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12d4685a-c83b-4010-8a94-7072edd46188-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9m2km\" (UID: \"12d4685a-c83b-4010-8a94-7072edd46188\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" Jul 07 00:00:14.560437 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.560397 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nncmj\" (UniqueName: \"kubernetes.io/projected/12d4685a-c83b-4010-8a94-7072edd46188-kube-api-access-nncmj\") pod \"cert-manager-operator-controller-manager-55c444bf86-9m2km\" (UID: \"12d4685a-c83b-4010-8a94-7072edd46188\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" Jul 07 00:00:14.678617 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.678578 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" Jul 07 00:00:14.838671 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:14.838634 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km"] Jul 07 00:00:14.842063 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:00:14.842034 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d4685a_c83b_4010_8a94_7072edd46188.slice/crio-5aff1f4daaf9bb4ff7356371f765f33a15d319ea4726faa7a814823dee601fb5 WatchSource:0}: Error finding container 5aff1f4daaf9bb4ff7356371f765f33a15d319ea4726faa7a814823dee601fb5: Status 404 returned error can't find the container with id 5aff1f4daaf9bb4ff7356371f765f33a15d319ea4726faa7a814823dee601fb5 Jul 07 00:00:15.704425 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:15.704392 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" event={"ID":"12d4685a-c83b-4010-8a94-7072edd46188","Type":"ContainerStarted","Data":"5aff1f4daaf9bb4ff7356371f765f33a15d319ea4726faa7a814823dee601fb5"} Jul 07 00:00:15.995038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:15.994957 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:16.065061 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:16.065026 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:20.791494 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:20.791454 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:20.917736 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:20.917691 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:21.477911 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.477868 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:21.578569 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.578521 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:21.622264 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.622236 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9fkpl"] Jul 07 00:00:21.625503 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.625486 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:21.628328 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.628307 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5rrbb\"" Jul 07 00:00:21.629387 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.629372 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 00:00:21.629455 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.629393 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 00:00:21.638709 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.638685 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9fkpl"] Jul 07 00:00:21.706019 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.705985 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2236935a-b33c-48f8-9b47-5ca62866b9d0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9fkpl\" (UID: \"2236935a-b33c-48f8-9b47-5ca62866b9d0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:21.706019 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.706023 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9f6x\" (UniqueName: \"kubernetes.io/projected/2236935a-b33c-48f8-9b47-5ca62866b9d0-kube-api-access-d9f6x\") pod \"cert-manager-webhook-6486ffd997-9fkpl\" (UID: \"2236935a-b33c-48f8-9b47-5ca62866b9d0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:21.731170 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.731081 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" event={"ID":"12d4685a-c83b-4010-8a94-7072edd46188","Type":"ContainerStarted","Data":"938c935ed18b79ef73d752504be18bec06081b8d0249b677bf4a8e495ac2c9d8"} Jul 07 00:00:21.756955 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.756899 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9m2km" podStartSLOduration=1.7736359369999999 podStartE2EDuration="7.756879859s" totalImagesPullingTime="5.983243922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:00:21.755352637 +0000 UTC m=+259.700938009" watchObservedRunningTime="2026-07-07 00:00:21.756879859 +0000 UTC m=+259.702465231" Jul 07 00:00:21.807478 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.807432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2236935a-b33c-48f8-9b47-5ca62866b9d0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9fkpl\" (UID: \"2236935a-b33c-48f8-9b47-5ca62866b9d0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:21.807854 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.807482 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9f6x\" (UniqueName: \"kubernetes.io/projected/2236935a-b33c-48f8-9b47-5ca62866b9d0-kube-api-access-d9f6x\") pod \"cert-manager-webhook-6486ffd997-9fkpl\" (UID: \"2236935a-b33c-48f8-9b47-5ca62866b9d0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:21.816413 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.816385 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9f6x\" (UniqueName: \"kubernetes.io/projected/2236935a-b33c-48f8-9b47-5ca62866b9d0-kube-api-access-d9f6x\") pod \"cert-manager-webhook-6486ffd997-9fkpl\" (UID: \"2236935a-b33c-48f8-9b47-5ca62866b9d0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:21.816664 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.816647 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2236935a-b33c-48f8-9b47-5ca62866b9d0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9fkpl\" (UID: \"2236935a-b33c-48f8-9b47-5ca62866b9d0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:21.944762 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:21.944722 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:22.082147 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:22.082117 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9fkpl"] Jul 07 00:00:22.085085 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:00:22.085055 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2236935a_b33c_48f8_9b47_5ca62866b9d0.slice/crio-777f146d71ead9c28901a1d3498fc292cf41af7293b20a757ca926a61ebbfe89 WatchSource:0}: Error finding container 777f146d71ead9c28901a1d3498fc292cf41af7293b20a757ca926a61ebbfe89: Status 404 returned error can't find the container with id 777f146d71ead9c28901a1d3498fc292cf41af7293b20a757ca926a61ebbfe89 Jul 07 00:00:22.184297 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:22.184241 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:22.384792 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:22.384701 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:22.736864 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:22.736828 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" event={"ID":"2236935a-b33c-48f8-9b47-5ca62866b9d0","Type":"ContainerStarted","Data":"777f146d71ead9c28901a1d3498fc292cf41af7293b20a757ca926a61ebbfe89"} Jul 07 00:00:23.611896 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.611855 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n"] Jul 07 00:00:23.614638 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.614614 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.617485 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.617274 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 00:00:23.617485 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.617357 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 00:00:23.618432 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.618411 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jrqh2\"" Jul 07 00:00:23.624774 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.624750 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n"] Jul 07 00:00:23.693253 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.693213 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:23.725632 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.725589 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.725810 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.725643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.725810 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.725745 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8qn\" (UniqueName: \"kubernetes.io/projected/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-kube-api-access-dl8qn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.826260 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.826216 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.826260 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.826279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.826508 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.826318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8qn\" (UniqueName: \"kubernetes.io/projected/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-kube-api-access-dl8qn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.826703 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.826683 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.826768 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.826750 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.835724 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.835692 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8qn\" (UniqueName: \"kubernetes.io/projected/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-kube-api-access-dl8qn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:23.927164 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:23.927126 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:24.049971 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:24.049939 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:24.730811 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:24.730782 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n"] Jul 07 00:00:24.733136 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:00:24.733103 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f0ccd6_4a5f_4a29_a16a_44f6af33037e.slice/crio-ddbba7afe02d2b474daf75064125afc8dcbc16a3c76401605d0a8554350a95a8 WatchSource:0}: Error finding container ddbba7afe02d2b474daf75064125afc8dcbc16a3c76401605d0a8554350a95a8: Status 404 returned error can't find the container with id ddbba7afe02d2b474daf75064125afc8dcbc16a3c76401605d0a8554350a95a8 Jul 07 00:00:24.747002 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:24.746965 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" event={"ID":"2236935a-b33c-48f8-9b47-5ca62866b9d0","Type":"ContainerStarted","Data":"ba04d3554d447dc3c421d06c1380621228eda021d2562cee756b881af90181e7"} Jul 07 00:00:24.747100 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:24.747081 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:24.748170 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:24.748147 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" event={"ID":"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e","Type":"ContainerStarted","Data":"ddbba7afe02d2b474daf75064125afc8dcbc16a3c76401605d0a8554350a95a8"} Jul 07 00:00:24.763599 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:24.763526 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" podStartSLOduration=1.2198169349999999 podStartE2EDuration="3.763509942s" totalImagesPullingTime="2.543693007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:00:24.763214463 +0000 UTC m=+262.708799846" watchObservedRunningTime="2026-07-07 00:00:24.763509942 +0000 UTC m=+262.709095313" Jul 07 00:00:25.756859 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:25.756819 2563 generic.go:356] "Generic (PLEG): container finished" podID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerID="dbb9a9c82d7a2b1a063bfc32ae68f934fb8101ebc7b4a1c5122a5df2b919e1b7" exitCode=0 Jul 07 00:00:25.757364 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:25.756903 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" event={"ID":"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e","Type":"ContainerDied","Data":"dbb9a9c82d7a2b1a063bfc32ae68f934fb8101ebc7b4a1c5122a5df2b919e1b7"} Jul 07 00:00:28.225899 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.225861 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w"] Jul 07 00:00:28.228298 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.228276 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" Jul 07 00:00:28.230703 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.230680 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-55ppc\"" Jul 07 00:00:28.239172 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.239146 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w"] Jul 07 00:00:28.367367 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.367330 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5cqt\" (UniqueName: \"kubernetes.io/projected/94782f14-d8da-4592-a21a-bd4f30c7023a-kube-api-access-b5cqt\") pod \"cert-manager-cainjector-559cdd8c87-tgd4w\" (UID: \"94782f14-d8da-4592-a21a-bd4f30c7023a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" Jul 07 00:00:28.367521 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.367402 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94782f14-d8da-4592-a21a-bd4f30c7023a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tgd4w\" (UID: \"94782f14-d8da-4592-a21a-bd4f30c7023a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" Jul 07 00:00:28.468558 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.468498 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94782f14-d8da-4592-a21a-bd4f30c7023a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tgd4w\" (UID: \"94782f14-d8da-4592-a21a-bd4f30c7023a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" Jul 07 00:00:28.468766 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.468612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5cqt\" (UniqueName: \"kubernetes.io/projected/94782f14-d8da-4592-a21a-bd4f30c7023a-kube-api-access-b5cqt\") pod \"cert-manager-cainjector-559cdd8c87-tgd4w\" (UID: \"94782f14-d8da-4592-a21a-bd4f30c7023a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" Jul 07 00:00:28.477040 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.476968 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94782f14-d8da-4592-a21a-bd4f30c7023a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tgd4w\" (UID: \"94782f14-d8da-4592-a21a-bd4f30c7023a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" Jul 07 00:00:28.477040 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.477020 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5cqt\" (UniqueName: \"kubernetes.io/projected/94782f14-d8da-4592-a21a-bd4f30c7023a-kube-api-access-b5cqt\") pod \"cert-manager-cainjector-559cdd8c87-tgd4w\" (UID: \"94782f14-d8da-4592-a21a-bd4f30c7023a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" Jul 07 00:00:28.537813 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.537779 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" Jul 07 00:00:28.674391 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.674367 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w"] Jul 07 00:00:28.676728 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:00:28.676700 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94782f14_d8da_4592_a21a_bd4f30c7023a.slice/crio-ee7f06caafb24b9b34fb905e5c681966045969297a1358604749c4718f27c0a0 WatchSource:0}: Error finding container ee7f06caafb24b9b34fb905e5c681966045969297a1358604749c4718f27c0a0: Status 404 returned error can't find the container with id ee7f06caafb24b9b34fb905e5c681966045969297a1358604749c4718f27c0a0 Jul 07 00:00:28.771154 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.771121 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" event={"ID":"94782f14-d8da-4592-a21a-bd4f30c7023a","Type":"ContainerStarted","Data":"6be1d72907a9786db43df367c2d57bcdbe68aadd6a2c42fe35787eed0b18b683"} Jul 07 00:00:28.771335 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.771172 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" event={"ID":"94782f14-d8da-4592-a21a-bd4f30c7023a","Type":"ContainerStarted","Data":"ee7f06caafb24b9b34fb905e5c681966045969297a1358604749c4718f27c0a0"} Jul 07 00:00:28.773150 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.773119 2563 generic.go:356] "Generic (PLEG): container finished" podID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerID="1b702bd9ebf9d857660f3b47fc45f6a864c6b363cfba5447b2d6b33e33bb7a51" exitCode=0 Jul 07 00:00:28.773337 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.773151 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" event={"ID":"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e","Type":"ContainerDied","Data":"1b702bd9ebf9d857660f3b47fc45f6a864c6b363cfba5447b2d6b33e33bb7a51"} Jul 07 00:00:28.786352 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:28.786304 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgd4w" podStartSLOduration=0.786288912 podStartE2EDuration="786.288912ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:00:28.786195827 +0000 UTC m=+266.731781198" watchObservedRunningTime="2026-07-07 00:00:28.786288912 +0000 UTC m=+266.731874283" Jul 07 00:00:29.456891 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:29.456846 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:29.734027 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:29.733947 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:29.779188 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:29.779156 2563 generic.go:356] "Generic (PLEG): container finished" podID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerID="3bf35e8f2618cc4eb1f6c4e15bc1090b61276135c6a6045e4abc11f6d794a6d8" exitCode=0 Jul 07 00:00:29.779355 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:29.779257 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" event={"ID":"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e","Type":"ContainerDied","Data":"3bf35e8f2618cc4eb1f6c4e15bc1090b61276135c6a6045e4abc11f6d794a6d8"} Jul 07 00:00:30.759835 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:30.759796 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9fkpl" Jul 07 00:00:30.922342 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:30.922318 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:30.991034 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:30.991002 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl8qn\" (UniqueName: \"kubernetes.io/projected/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-kube-api-access-dl8qn\") pod \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " Jul 07 00:00:30.991208 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:30.991051 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-bundle\") pod \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " Jul 07 00:00:30.991208 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:30.991080 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-util\") pod \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\" (UID: \"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e\") " Jul 07 00:00:30.991489 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:30.991453 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-bundle" (OuterVolumeSpecName: "bundle") pod "e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" (UID: "e8f0ccd6-4a5f-4a29-a16a-44f6af33037e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:00:30.993379 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:30.993349 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-kube-api-access-dl8qn" (OuterVolumeSpecName: "kube-api-access-dl8qn") pod "e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" (UID: "e8f0ccd6-4a5f-4a29-a16a-44f6af33037e"). InnerVolumeSpecName "kube-api-access-dl8qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:00:30.995852 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:30.995823 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-util" (OuterVolumeSpecName: "util") pod "e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" (UID: "e8f0ccd6-4a5f-4a29-a16a-44f6af33037e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:00:31.092523 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:31.092422 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl8qn\" (UniqueName: \"kubernetes.io/projected/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-kube-api-access-dl8qn\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:31.092523 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:31.092461 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-bundle\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:31.092523 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:31.092476 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8f0ccd6-4a5f-4a29-a16a-44f6af33037e-util\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:31.790751 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:31.790712 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" Jul 07 00:00:31.790751 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:31.790727 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqw72n" event={"ID":"e8f0ccd6-4a5f-4a29-a16a-44f6af33037e","Type":"ContainerDied","Data":"ddbba7afe02d2b474daf75064125afc8dcbc16a3c76401605d0a8554350a95a8"} Jul 07 00:00:31.791276 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:31.790771 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddbba7afe02d2b474daf75064125afc8dcbc16a3c76401605d0a8554350a95a8" Jul 07 00:00:32.054807 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:32.054719 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:39.342649 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:39.342609 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:41.024663 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:41.024631 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:45.417520 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:45.417489 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:45.517972 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:45.517927 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:47.445869 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:47.445824 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:48.269665 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.269635 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl"] Jul 07 00:00:48.269980 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.269963 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerName="pull" Jul 07 00:00:48.270059 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.269982 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerName="pull" Jul 07 00:00:48.270059 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.270008 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerName="extract" Jul 07 00:00:48.270059 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.270017 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerName="extract" Jul 07 00:00:48.270059 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.270030 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerName="util" Jul 07 00:00:48.270059 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.270038 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerName="util" Jul 07 00:00:48.270276 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.270123 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f0ccd6-4a5f-4a29-a16a-44f6af33037e" containerName="extract" Jul 07 00:00:48.272700 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.272676 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.275381 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.275346 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 00:00:48.276334 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.276316 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jrqh2\"" Jul 07 00:00:48.276413 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.276337 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 00:00:48.281777 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.281752 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl"] Jul 07 00:00:48.435776 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.435741 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.435963 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.435788 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.435963 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.435837 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzfp\" (UniqueName: \"kubernetes.io/projected/9b12398e-a54a-480d-a75b-4c97df5013fb-kube-api-access-xtzfp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.536903 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.536813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.536903 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.536866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.537293 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.536905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzfp\" (UniqueName: \"kubernetes.io/projected/9b12398e-a54a-480d-a75b-4c97df5013fb-kube-api-access-xtzfp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.537293 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.537198 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.537293 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.537227 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.545336 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.545302 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzfp\" (UniqueName: \"kubernetes.io/projected/9b12398e-a54a-480d-a75b-4c97df5013fb-kube-api-access-xtzfp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.582490 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.582453 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:48.726142 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.726007 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl"] Jul 07 00:00:48.729072 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:00:48.729045 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b12398e_a54a_480d_a75b_4c97df5013fb.slice/crio-63befeb169b85f6d34136aa3562aa697c1f7ec12688ddab1564d7da8342c697b WatchSource:0}: Error finding container 63befeb169b85f6d34136aa3562aa697c1f7ec12688ddab1564d7da8342c697b: Status 404 returned error can't find the container with id 63befeb169b85f6d34136aa3562aa697c1f7ec12688ddab1564d7da8342c697b Jul 07 00:00:48.864550 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.864505 2563 generic.go:356] "Generic (PLEG): container finished" podID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerID="0f10305b1d5db02118b3435aab7fb14b6adcf70e7c759a85110f9c56cca61f2c" exitCode=0 Jul 07 00:00:48.864720 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.864577 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" event={"ID":"9b12398e-a54a-480d-a75b-4c97df5013fb","Type":"ContainerDied","Data":"0f10305b1d5db02118b3435aab7fb14b6adcf70e7c759a85110f9c56cca61f2c"} Jul 07 00:00:48.864720 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:48.864633 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" event={"ID":"9b12398e-a54a-480d-a75b-4c97df5013fb","Type":"ContainerStarted","Data":"63befeb169b85f6d34136aa3562aa697c1f7ec12688ddab1564d7da8342c697b"} Jul 07 00:00:50.875010 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:50.874975 2563 generic.go:356] "Generic (PLEG): container finished" podID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerID="ec56c85cbf6a6cecc022831eb06e49a8465675592efcc2c462c5849c09ff20ad" exitCode=0 Jul 07 00:00:50.875468 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:50.875052 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" event={"ID":"9b12398e-a54a-480d-a75b-4c97df5013fb","Type":"ContainerDied","Data":"ec56c85cbf6a6cecc022831eb06e49a8465675592efcc2c462c5849c09ff20ad"} Jul 07 00:00:51.881471 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:51.881436 2563 generic.go:356] "Generic (PLEG): container finished" podID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerID="d3eeb31ac2a4fde6cb3b2cff0cc8a934a12859bac3135e24ab7317547c24e6ec" exitCode=0 Jul 07 00:00:51.882006 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:51.881493 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" event={"ID":"9b12398e-a54a-480d-a75b-4c97df5013fb","Type":"ContainerDied","Data":"d3eeb31ac2a4fde6cb3b2cff0cc8a934a12859bac3135e24ab7317547c24e6ec"} Jul 07 00:00:53.010997 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.010967 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:53.180492 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.180450 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-bundle\") pod \"9b12398e-a54a-480d-a75b-4c97df5013fb\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " Jul 07 00:00:53.180492 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.180508 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtzfp\" (UniqueName: \"kubernetes.io/projected/9b12398e-a54a-480d-a75b-4c97df5013fb-kube-api-access-xtzfp\") pod \"9b12398e-a54a-480d-a75b-4c97df5013fb\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " Jul 07 00:00:53.180781 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.180588 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-util\") pod \"9b12398e-a54a-480d-a75b-4c97df5013fb\" (UID: \"9b12398e-a54a-480d-a75b-4c97df5013fb\") " Jul 07 00:00:53.181323 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.181271 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-bundle" (OuterVolumeSpecName: "bundle") pod "9b12398e-a54a-480d-a75b-4c97df5013fb" (UID: "9b12398e-a54a-480d-a75b-4c97df5013fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:00:53.182847 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.182822 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b12398e-a54a-480d-a75b-4c97df5013fb-kube-api-access-xtzfp" (OuterVolumeSpecName: "kube-api-access-xtzfp") pod "9b12398e-a54a-480d-a75b-4c97df5013fb" (UID: "9b12398e-a54a-480d-a75b-4c97df5013fb"). InnerVolumeSpecName "kube-api-access-xtzfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:00:53.186199 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.186174 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-util" (OuterVolumeSpecName: "util") pod "9b12398e-a54a-480d-a75b-4c97df5013fb" (UID: "9b12398e-a54a-480d-a75b-4c97df5013fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:00:53.281303 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.281253 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-util\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:53.281303 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.281297 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b12398e-a54a-480d-a75b-4c97df5013fb-bundle\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:53.281303 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.281307 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtzfp\" (UniqueName: \"kubernetes.io/projected/9b12398e-a54a-480d-a75b-4c97df5013fb-kube-api-access-xtzfp\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:00:53.891825 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.891792 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" event={"ID":"9b12398e-a54a-480d-a75b-4c97df5013fb","Type":"ContainerDied","Data":"63befeb169b85f6d34136aa3562aa697c1f7ec12688ddab1564d7da8342c697b"} Jul 07 00:00:53.892004 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.891839 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63befeb169b85f6d34136aa3562aa697c1f7ec12688ddab1564d7da8342c697b" Jul 07 00:00:53.892004 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:53.891807 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7vnl" Jul 07 00:00:59.414959 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:59.414920 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:00:59.511805 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:00:59.511769 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:00.020698 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.020662 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:00.382120 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.382040 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:00.483160 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.483133 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:00.514362 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.514330 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d"] Jul 07 00:01:00.514696 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.514682 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerName="extract" Jul 07 00:01:00.514744 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.514700 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerName="extract" Jul 07 00:01:00.514744 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.514713 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerName="util" Jul 07 00:01:00.514744 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.514718 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerName="util" Jul 07 00:01:00.514744 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.514728 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerName="pull" Jul 07 00:01:00.514744 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.514734 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerName="pull" Jul 07 00:01:00.514927 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.514784 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b12398e-a54a-480d-a75b-4c97df5013fb" containerName="extract" Jul 07 00:01:00.519180 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.519159 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.524020 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.523994 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 00:01:00.524194 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.524058 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 00:01:00.524663 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.524638 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 00:01:00.524809 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.524760 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 00:01:00.524952 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.524930 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-swm29\"" Jul 07 00:01:00.534105 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.534077 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-manifests\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.534252 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.534119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.534252 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.534211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrrw7\" (UniqueName: \"kubernetes.io/projected/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-kube-api-access-zrrw7\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.534359 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.534249 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-tmp\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.534359 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.534283 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.539491 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.539460 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d"] Jul 07 00:01:00.634989 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.634887 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.634989 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.634964 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrrw7\" (UniqueName: \"kubernetes.io/projected/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-kube-api-access-zrrw7\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.635227 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.634992 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-tmp\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.635227 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.635022 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.635227 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.635051 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-manifests\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.635496 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.635473 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-manifests\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.637460 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.637437 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-tmp\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.637654 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.637635 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.637710 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.637672 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.653903 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.653873 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrrw7\" (UniqueName: \"kubernetes.io/projected/e7a3bbe0-438d-4f5a-84de-51b268f2d2bf-kube-api-access-zrrw7\") pod \"opendatahub-operator-controller-manager-7c7bd8fcf-qft7d\" (UID: \"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.686136 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.686098 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:00.786250 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.786221 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:00.830242 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.830195 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:00.885120 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.885055 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:00.996291 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.996267 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d"] Jul 07 00:01:00.997236 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:00.997207 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:00.998574 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:01:00.998518 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a3bbe0_438d_4f5a_84de_51b268f2d2bf.slice/crio-81f3fab31cdcc1865cdb13c506fd0fe997b458c7f080d966e124fc079407067c WatchSource:0}: Error finding container 81f3fab31cdcc1865cdb13c506fd0fe997b458c7f080d966e124fc079407067c: Status 404 returned error can't find the container with id 81f3fab31cdcc1865cdb13c506fd0fe997b458c7f080d966e124fc079407067c Jul 07 00:01:01.101706 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:01.101673 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:01.930483 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:01.930441 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" event={"ID":"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf","Type":"ContainerStarted","Data":"81f3fab31cdcc1865cdb13c506fd0fe997b458c7f080d966e124fc079407067c"} Jul 07 00:01:02.462056 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:02.462030 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 00:01:05.952777 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:05.952743 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" event={"ID":"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf","Type":"ContainerStarted","Data":"24eb295d32d9a34c536e2817bdca64f2f030adfe094adac303317d8c7a791601"} Jul 07 00:01:06.959226 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:06.959190 2563 generic.go:356] "Generic (PLEG): container finished" podID="e7a3bbe0-438d-4f5a-84de-51b268f2d2bf" containerID="24eb295d32d9a34c536e2817bdca64f2f030adfe094adac303317d8c7a791601" exitCode=0 Jul 07 00:01:06.959723 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:06.959273 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" event={"ID":"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf","Type":"ContainerDied","Data":"24eb295d32d9a34c536e2817bdca64f2f030adfe094adac303317d8c7a791601"} Jul 07 00:01:06.959917 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:06.959898 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:01:07.965609 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:07.965572 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" event={"ID":"e7a3bbe0-438d-4f5a-84de-51b268f2d2bf","Type":"ContainerStarted","Data":"787d9d8e13985381f7f0d1b663bb81e01d7b75ac7dd515da6af3103ab88d2ee6"} Jul 07 00:01:07.965985 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:07.965709 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:07.990813 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:07.990763 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" podStartSLOduration=1.8500021549999999 podStartE2EDuration="7.990744024s" totalImagesPullingTime="5.063382202s" totalInitContainerRuntime="1.077359667s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:07.988658782 +0000 UTC m=+305.934244156" watchObservedRunningTime="2026-07-07 00:01:07.990744024 +0000 UTC m=+305.936329395" Jul 07 00:01:09.723815 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:09.723780 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:09.827122 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:09.827091 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:10.477570 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:10.477522 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:10.547951 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:10.547906 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:18.973360 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:18.973328 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7bd8fcf-qft7d" Jul 07 00:01:20.395199 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.395170 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8"] Jul 07 00:01:20.403291 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.403269 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.407582 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.407519 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z9lxp\"" Jul 07 00:01:20.407730 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.407702 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 00:01:20.408453 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.408435 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 00:01:20.408661 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.408637 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 00:01:20.408661 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.408653 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 00:01:20.408838 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.408674 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 00:01:20.413903 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.413877 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8"] Jul 07 00:01:20.501159 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.501118 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb2d6a38-3149-427d-aa83-beb38726ea43-cert\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.501324 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.501199 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbr8\" (UniqueName: \"kubernetes.io/projected/eb2d6a38-3149-427d-aa83-beb38726ea43-kube-api-access-scbr8\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.501324 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.501264 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eb2d6a38-3149-427d-aa83-beb38726ea43-manager-config\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.501324 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.501314 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb2d6a38-3149-427d-aa83-beb38726ea43-metrics-cert\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.602221 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.602182 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbr8\" (UniqueName: \"kubernetes.io/projected/eb2d6a38-3149-427d-aa83-beb38726ea43-kube-api-access-scbr8\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.602221 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.602226 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eb2d6a38-3149-427d-aa83-beb38726ea43-manager-config\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.602485 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.602248 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb2d6a38-3149-427d-aa83-beb38726ea43-metrics-cert\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.602485 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.602279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb2d6a38-3149-427d-aa83-beb38726ea43-cert\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.602935 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.602910 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eb2d6a38-3149-427d-aa83-beb38726ea43-manager-config\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.604998 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.604967 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb2d6a38-3149-427d-aa83-beb38726ea43-cert\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.604998 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.604994 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb2d6a38-3149-427d-aa83-beb38726ea43-metrics-cert\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.611195 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.611167 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:20.613729 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.613704 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbr8\" (UniqueName: \"kubernetes.io/projected/eb2d6a38-3149-427d-aa83-beb38726ea43-kube-api-access-scbr8\") pod \"lws-controller-manager-6647df896d-w4zh8\" (UID: \"eb2d6a38-3149-427d-aa83-beb38726ea43\") " pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.712920 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.712834 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:20.719577 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.719521 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:20.881009 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:01:20.880944 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2d6a38_3149_427d_aa83_beb38726ea43.slice/crio-54db9becb3e0807319a700c3525b03fa34fc42c21d6493d1833b56b0e45bd37b WatchSource:0}: Error finding container 54db9becb3e0807319a700c3525b03fa34fc42c21d6493d1833b56b0e45bd37b: Status 404 returned error can't find the container with id 54db9becb3e0807319a700c3525b03fa34fc42c21d6493d1833b56b0e45bd37b Jul 07 00:01:20.881277 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:20.881257 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8"] Jul 07 00:01:21.026845 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.026754 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" event={"ID":"eb2d6a38-3149-427d-aa83-beb38726ea43","Type":"ContainerStarted","Data":"54db9becb3e0807319a700c3525b03fa34fc42c21d6493d1833b56b0e45bd37b"} Jul 07 00:01:21.052061 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.052034 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:21.110257 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.110224 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9"] Jul 07 00:01:21.115886 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.115358 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.119501 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.119462 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 00:01:21.119745 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.119716 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6kr5x\"" Jul 07 00:01:21.120112 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.120091 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 00:01:21.120214 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.120091 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 00:01:21.121837 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.121818 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 00:01:21.125582 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.125557 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9"] Jul 07 00:01:21.208812 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.208781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de610c6-bd2a-4956-8026-25c882cfa4f2-tmp\") pod \"kube-auth-proxy-6b5dbc58cc-7t9k9\" (UID: \"4de610c6-bd2a-4956-8026-25c882cfa4f2\") " pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.209000 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.208843 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzn4\" (UniqueName: \"kubernetes.io/projected/4de610c6-bd2a-4956-8026-25c882cfa4f2-kube-api-access-5vzn4\") pod \"kube-auth-proxy-6b5dbc58cc-7t9k9\" (UID: \"4de610c6-bd2a-4956-8026-25c882cfa4f2\") " pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.209000 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.208931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de610c6-bd2a-4956-8026-25c882cfa4f2-tls-certs\") pod \"kube-auth-proxy-6b5dbc58cc-7t9k9\" (UID: \"4de610c6-bd2a-4956-8026-25c882cfa4f2\") " pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.267569 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.267510 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:21.309954 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.309866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzn4\" (UniqueName: \"kubernetes.io/projected/4de610c6-bd2a-4956-8026-25c882cfa4f2-kube-api-access-5vzn4\") pod \"kube-auth-proxy-6b5dbc58cc-7t9k9\" (UID: \"4de610c6-bd2a-4956-8026-25c882cfa4f2\") " pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.309954 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.309911 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de610c6-bd2a-4956-8026-25c882cfa4f2-tls-certs\") pod \"kube-auth-proxy-6b5dbc58cc-7t9k9\" (UID: \"4de610c6-bd2a-4956-8026-25c882cfa4f2\") " pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.309954 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.309946 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de610c6-bd2a-4956-8026-25c882cfa4f2-tmp\") pod \"kube-auth-proxy-6b5dbc58cc-7t9k9\" (UID: \"4de610c6-bd2a-4956-8026-25c882cfa4f2\") " pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.312343 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.312316 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de610c6-bd2a-4956-8026-25c882cfa4f2-tmp\") pod \"kube-auth-proxy-6b5dbc58cc-7t9k9\" (UID: \"4de610c6-bd2a-4956-8026-25c882cfa4f2\") " pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.312556 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.312520 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de610c6-bd2a-4956-8026-25c882cfa4f2-tls-certs\") pod \"kube-auth-proxy-6b5dbc58cc-7t9k9\" (UID: \"4de610c6-bd2a-4956-8026-25c882cfa4f2\") " pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.320513 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.320485 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzn4\" (UniqueName: \"kubernetes.io/projected/4de610c6-bd2a-4956-8026-25c882cfa4f2-kube-api-access-5vzn4\") pod \"kube-auth-proxy-6b5dbc58cc-7t9k9\" (UID: \"4de610c6-bd2a-4956-8026-25c882cfa4f2\") " pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.420468 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.420431 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:21.427244 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.427213 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" Jul 07 00:01:21.538304 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.538256 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:21.597059 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.596930 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9"] Jul 07 00:01:21.599483 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:01:21.599441 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de610c6_bd2a_4956_8026_25c882cfa4f2.slice/crio-763aea0da39f65d8b6b1eca2bfde37abfe55155c12019002779f98ac6f4ea3e0 WatchSource:0}: Error finding container 763aea0da39f65d8b6b1eca2bfde37abfe55155c12019002779f98ac6f4ea3e0: Status 404 returned error can't find the container with id 763aea0da39f65d8b6b1eca2bfde37abfe55155c12019002779f98ac6f4ea3e0 Jul 07 00:01:21.659978 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:21.659941 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:22.032986 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:22.032947 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" event={"ID":"4de610c6-bd2a-4956-8026-25c882cfa4f2","Type":"ContainerStarted","Data":"763aea0da39f65d8b6b1eca2bfde37abfe55155c12019002779f98ac6f4ea3e0"} Jul 07 00:01:22.602978 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:22.602825 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:22.929020 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:22.928981 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:23.039767 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:23.039697 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" event={"ID":"eb2d6a38-3149-427d-aa83-beb38726ea43","Type":"ContainerStarted","Data":"6d1c327e7a5facc071057b7250bf97e7ea8a7b735594375c5713145ca1cc6346"} Jul 07 00:01:23.039961 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:23.039792 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:23.069743 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:23.069683 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" podStartSLOduration=1.741710619 podStartE2EDuration="3.069665142s" totalImagesPullingTime="1.327954523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:23.068757082 +0000 UTC m=+321.014342455" watchObservedRunningTime="2026-07-07 00:01:23.069665142 +0000 UTC m=+321.015250512" Jul 07 00:01:23.181459 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:23.181375 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:23.288588 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:23.288525 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:24.045590 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:24.045549 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" event={"ID":"4de610c6-bd2a-4956-8026-25c882cfa4f2","Type":"ContainerStarted","Data":"ae39a81f4b8cddc7cf4698db2fd1f7f8f3ac7d08b5bd78cfb7875042212433f3"} Jul 07 00:01:24.066508 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:24.066443 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b5dbc58cc-7t9k9" podStartSLOduration=0.841687626 podStartE2EDuration="3.066424635s" totalImagesPullingTime="2.224737009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:24.06604719 +0000 UTC m=+322.011632563" watchObservedRunningTime="2026-07-07 00:01:24.066424635 +0000 UTC m=+322.012010006" Jul 07 00:01:24.679827 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:24.679790 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:28.632076 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:28.632029 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:28.780819 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:28.780785 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:29.136391 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:29.136361 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:29.530833 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:29.530797 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:29.861276 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:29.861198 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:34.048615 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:34.048575 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6647df896d-w4zh8" Jul 07 00:01:35.849111 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:35.849080 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:36.003739 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.003700 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:36.150253 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.150214 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5"] Jul 07 00:01:36.156486 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.156454 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:36.161838 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.161809 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 07 00:01:36.161998 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.161843 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dptrp\"" Jul 07 00:01:36.198587 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.198553 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5"] Jul 07 00:01:36.248044 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.248008 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e56473a-90eb-48a1-b8eb-63d574565722-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-btcv5\" (UID: \"6e56473a-90eb-48a1-b8eb-63d574565722\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:36.248231 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.248082 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjfs\" (UniqueName: \"kubernetes.io/projected/6e56473a-90eb-48a1-b8eb-63d574565722-kube-api-access-nrjfs\") pod \"llmisvc-controller-manager-777c6bd7d5-btcv5\" (UID: \"6e56473a-90eb-48a1-b8eb-63d574565722\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:36.254482 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.254440 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:36.349426 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.349381 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjfs\" (UniqueName: \"kubernetes.io/projected/6e56473a-90eb-48a1-b8eb-63d574565722-kube-api-access-nrjfs\") pod \"llmisvc-controller-manager-777c6bd7d5-btcv5\" (UID: \"6e56473a-90eb-48a1-b8eb-63d574565722\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:36.349669 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.349447 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e56473a-90eb-48a1-b8eb-63d574565722-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-btcv5\" (UID: \"6e56473a-90eb-48a1-b8eb-63d574565722\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:36.349669 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:01:36.349593 2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 07 00:01:36.349669 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:01:36.349663 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e56473a-90eb-48a1-b8eb-63d574565722-cert podName:6e56473a-90eb-48a1-b8eb-63d574565722 nodeName:}" failed. No retries permitted until 2026-07-07 00:01:36.84964785 +0000 UTC m=+334.795233200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6e56473a-90eb-48a1-b8eb-63d574565722-cert") pod "llmisvc-controller-manager-777c6bd7d5-btcv5" (UID: "6e56473a-90eb-48a1-b8eb-63d574565722") : secret "llmisvc-webhook-server-cert" not found Jul 07 00:01:36.404497 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.404422 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjfs\" (UniqueName: \"kubernetes.io/projected/6e56473a-90eb-48a1-b8eb-63d574565722-kube-api-access-nrjfs\") pod \"llmisvc-controller-manager-777c6bd7d5-btcv5\" (UID: \"6e56473a-90eb-48a1-b8eb-63d574565722\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:36.471783 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.471741 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:36.720407 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.720322 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:36.853520 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.853482 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e56473a-90eb-48a1-b8eb-63d574565722-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-btcv5\" (UID: \"6e56473a-90eb-48a1-b8eb-63d574565722\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:36.853959 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:01:36.853638 2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 07 00:01:36.853959 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:01:36.853714 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e56473a-90eb-48a1-b8eb-63d574565722-cert podName:6e56473a-90eb-48a1-b8eb-63d574565722 nodeName:}" failed. No retries permitted until 2026-07-07 00:01:37.853698477 +0000 UTC m=+335.799283827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6e56473a-90eb-48a1-b8eb-63d574565722-cert") pod "llmisvc-controller-manager-777c6bd7d5-btcv5" (UID: "6e56473a-90eb-48a1-b8eb-63d574565722") : secret "llmisvc-webhook-server-cert" not found Jul 07 00:01:36.907798 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:36.907756 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:37.120920 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:37.120828 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:01:37.861820 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:37.861783 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e56473a-90eb-48a1-b8eb-63d574565722-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-btcv5\" (UID: \"6e56473a-90eb-48a1-b8eb-63d574565722\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:37.864351 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:37.864317 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e56473a-90eb-48a1-b8eb-63d574565722-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-btcv5\" (UID: \"6e56473a-90eb-48a1-b8eb-63d574565722\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:37.966558 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:37.966488 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:38.134494 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:38.134307 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5"] Jul 07 00:01:38.137128 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:01:38.137098 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6e56473a_90eb_48a1_b8eb_63d574565722.slice/crio-54ac5b9dde0edcf3db7b61bb3f2180380ab61b2eb537a0e839dfc8f01b4d52f2 WatchSource:0}: Error finding container 54ac5b9dde0edcf3db7b61bb3f2180380ab61b2eb537a0e839dfc8f01b4d52f2: Status 404 returned error can't find the container with id 54ac5b9dde0edcf3db7b61bb3f2180380ab61b2eb537a0e839dfc8f01b4d52f2 Jul 07 00:01:39.122962 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:39.122924 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" event={"ID":"6e56473a-90eb-48a1-b8eb-63d574565722","Type":"ContainerStarted","Data":"54ac5b9dde0edcf3db7b61bb3f2180380ab61b2eb537a0e839dfc8f01b4d52f2"} Jul 07 00:01:41.133609 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:41.133571 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" event={"ID":"6e56473a-90eb-48a1-b8eb-63d574565722","Type":"ContainerStarted","Data":"43105c7b19a615caa2decaa3646edd6061c59188a556be97dd897e0ad40f615b"} Jul 07 00:01:41.134064 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:41.133687 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:01:44.108901 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:01:44.108866 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:02:12.140096 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:12.140060 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" Jul 07 00:02:12.159021 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:12.158972 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btcv5" podStartSLOduration=33.676785187 podStartE2EDuration="36.15894484s" totalImagesPullingTime="2.482159653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:01:41.192451355 +0000 UTC m=+339.138036728" watchObservedRunningTime="2026-07-07 00:02:12.15894484 +0000 UTC m=+370.104530211" Jul 07 00:02:22.115840 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:22.115809 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:02:22.216681 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:22.216644 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:02:22.358676 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:22.358623 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:02:37.378705 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.378666 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5"] Jul 07 00:02:37.382634 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.382601 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.385148 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.385124 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 00:02:37.386003 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.385988 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 00:02:37.386068 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.385989 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-htsf7\"" Jul 07 00:02:37.390378 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.390353 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5"] Jul 07 00:02:37.428775 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.428736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j969d\" (UniqueName: \"kubernetes.io/projected/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-kube-api-access-j969d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.428953 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.428807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.428953 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.428872 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.529798 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.529765 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j969d\" (UniqueName: \"kubernetes.io/projected/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-kube-api-access-j969d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.530001 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.529840 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.530001 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.529867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.530220 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.530199 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.530289 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.530249 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.538072 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.538045 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j969d\" (UniqueName: \"kubernetes.io/projected/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-kube-api-access-j969d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.692942 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.692912 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:37.825604 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.825579 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5"] Jul 07 00:02:37.828170 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:02:37.828139 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6136da15_a8e7_42a8_a387_1f4c7cd74ff7.slice/crio-fe3798cf25a9059270661861f2e987467473517ac5f1a60e1ae79246c09d9f8a WatchSource:0}: Error finding container fe3798cf25a9059270661861f2e987467473517ac5f1a60e1ae79246c09d9f8a: Status 404 returned error can't find the container with id fe3798cf25a9059270661861f2e987467473517ac5f1a60e1ae79246c09d9f8a Jul 07 00:02:37.979094 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.979018 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k"] Jul 07 00:02:37.981501 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.981480 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:37.991175 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:37.991140 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k"] Jul 07 00:02:38.034757 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.034717 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.034941 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.034780 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prlbk\" (UniqueName: \"kubernetes.io/projected/07956dfa-fedf-47cb-8211-9768035b3c00-kube-api-access-prlbk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.034941 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.034837 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.135405 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.135370 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.135603 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.135427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prlbk\" (UniqueName: \"kubernetes.io/projected/07956dfa-fedf-47cb-8211-9768035b3c00-kube-api-access-prlbk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.135603 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.135463 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.135825 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.135793 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.135861 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.135844 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.144202 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.144175 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prlbk\" (UniqueName: \"kubernetes.io/projected/07956dfa-fedf-47cb-8211-9768035b3c00-kube-api-access-prlbk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.294647 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.294561 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:38.384260 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.384220 2563 generic.go:356] "Generic (PLEG): container finished" podID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerID="ecb7ad3699bd0cff3da397db62f238e34ef482bbabe39fe4c28b2af8762a20f6" exitCode=0 Jul 07 00:02:38.384714 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.384354 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" event={"ID":"6136da15-a8e7-42a8-a387-1f4c7cd74ff7","Type":"ContainerDied","Data":"ecb7ad3699bd0cff3da397db62f238e34ef482bbabe39fe4c28b2af8762a20f6"} Jul 07 00:02:38.384714 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.384407 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" event={"ID":"6136da15-a8e7-42a8-a387-1f4c7cd74ff7","Type":"ContainerStarted","Data":"fe3798cf25a9059270661861f2e987467473517ac5f1a60e1ae79246c09d9f8a"} Jul 07 00:02:38.430393 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.430371 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k"] Jul 07 00:02:38.432076 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:02:38.432049 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07956dfa_fedf_47cb_8211_9768035b3c00.slice/crio-6d1b78757899524dad311d46496ea3a491269d1893167d6f8a163a38356b784f WatchSource:0}: Error finding container 6d1b78757899524dad311d46496ea3a491269d1893167d6f8a163a38356b784f: Status 404 returned error can't find the container with id 6d1b78757899524dad311d46496ea3a491269d1893167d6f8a163a38356b784f Jul 07 00:02:38.579590 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.579296 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs"] Jul 07 00:02:38.582010 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.581988 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.590484 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.590457 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs"] Jul 07 00:02:38.639507 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.639469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bm5\" (UniqueName: \"kubernetes.io/projected/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-kube-api-access-q2bm5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.639507 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.639511 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.639763 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.639649 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.740088 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.740045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.740276 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.740125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bm5\" (UniqueName: \"kubernetes.io/projected/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-kube-api-access-q2bm5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.740276 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.740158 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.740478 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.740449 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.740595 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.740499 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.748903 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.748877 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bm5\" (UniqueName: \"kubernetes.io/projected/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-kube-api-access-q2bm5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:38.891583 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:38.891465 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:39.029185 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.029147 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs"] Jul 07 00:02:39.034082 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:02:39.033909 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9c918f_0b3e_45ef_a1c7_c6e39bbd0376.slice/crio-058b2f980261520c7fdcc44ea2f7e6434c210872bf2052c893089e65f1824f25 WatchSource:0}: Error finding container 058b2f980261520c7fdcc44ea2f7e6434c210872bf2052c893089e65f1824f25: Status 404 returned error can't find the container with id 058b2f980261520c7fdcc44ea2f7e6434c210872bf2052c893089e65f1824f25 Jul 07 00:02:39.181325 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.181286 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw"] Jul 07 00:02:39.185087 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.185061 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.193719 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.193679 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw"] Jul 07 00:02:39.244286 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.244254 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.244447 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.244321 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz8c2\" (UniqueName: \"kubernetes.io/projected/50991f3d-a1cf-4c51-a611-1f9c136f7749-kube-api-access-xz8c2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.244447 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.244368 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.345265 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.345231 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.345459 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.345292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz8c2\" (UniqueName: \"kubernetes.io/projected/50991f3d-a1cf-4c51-a611-1f9c136f7749-kube-api-access-xz8c2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.345459 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.345333 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.345748 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.345729 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.345796 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.345776 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.354163 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.354138 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz8c2\" (UniqueName: \"kubernetes.io/projected/50991f3d-a1cf-4c51-a611-1f9c136f7749-kube-api-access-xz8c2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.390865 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.390828 2563 generic.go:356] "Generic (PLEG): container finished" podID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerID="b2b54437022e141a5e356a4a812617215d453977942b0cb117998c8f14b3ea46" exitCode=0 Jul 07 00:02:39.391277 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.390909 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" event={"ID":"6136da15-a8e7-42a8-a387-1f4c7cd74ff7","Type":"ContainerDied","Data":"b2b54437022e141a5e356a4a812617215d453977942b0cb117998c8f14b3ea46"} Jul 07 00:02:39.392561 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.392520 2563 generic.go:356] "Generic (PLEG): container finished" podID="07956dfa-fedf-47cb-8211-9768035b3c00" containerID="1d7c2088910067b1c801a8d88232ecd3461ab087cddcc353dd8298ab192b872b" exitCode=0 Jul 07 00:02:39.392652 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.392577 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" event={"ID":"07956dfa-fedf-47cb-8211-9768035b3c00","Type":"ContainerDied","Data":"1d7c2088910067b1c801a8d88232ecd3461ab087cddcc353dd8298ab192b872b"} Jul 07 00:02:39.392652 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.392625 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" event={"ID":"07956dfa-fedf-47cb-8211-9768035b3c00","Type":"ContainerStarted","Data":"6d1b78757899524dad311d46496ea3a491269d1893167d6f8a163a38356b784f"} Jul 07 00:02:39.394172 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.394145 2563 generic.go:356] "Generic (PLEG): container finished" podID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerID="ee09c06a22fac2c773502ba8218c8a07ba7f42001d18b3d769f52d2c45808790" exitCode=0 Jul 07 00:02:39.394263 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.394206 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" event={"ID":"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376","Type":"ContainerDied","Data":"ee09c06a22fac2c773502ba8218c8a07ba7f42001d18b3d769f52d2c45808790"} Jul 07 00:02:39.394263 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.394249 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" event={"ID":"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376","Type":"ContainerStarted","Data":"058b2f980261520c7fdcc44ea2f7e6434c210872bf2052c893089e65f1824f25"} Jul 07 00:02:39.496125 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.496092 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:39.654041 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:39.654017 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw"] Jul 07 00:02:39.656178 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:02:39.656152 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50991f3d_a1cf_4c51_a611_1f9c136f7749.slice/crio-98b34a524a54a3c26c9d70948d564855b8241085177045072ac3f3a17d39b59b WatchSource:0}: Error finding container 98b34a524a54a3c26c9d70948d564855b8241085177045072ac3f3a17d39b59b: Status 404 returned error can't find the container with id 98b34a524a54a3c26c9d70948d564855b8241085177045072ac3f3a17d39b59b Jul 07 00:02:40.399726 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:40.399693 2563 generic.go:356] "Generic (PLEG): container finished" podID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerID="a53efa0ed4ae23d047a34a1fb1c497c7da23b2d8c3530ca5fcc6a9bd9cceedd0" exitCode=0 Jul 07 00:02:40.400145 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:40.399761 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" event={"ID":"50991f3d-a1cf-4c51-a611-1f9c136f7749","Type":"ContainerDied","Data":"a53efa0ed4ae23d047a34a1fb1c497c7da23b2d8c3530ca5fcc6a9bd9cceedd0"} Jul 07 00:02:40.400145 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:40.399801 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" event={"ID":"50991f3d-a1cf-4c51-a611-1f9c136f7749","Type":"ContainerStarted","Data":"98b34a524a54a3c26c9d70948d564855b8241085177045072ac3f3a17d39b59b"} Jul 07 00:02:40.401723 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:40.401700 2563 generic.go:356] "Generic (PLEG): container finished" podID="07956dfa-fedf-47cb-8211-9768035b3c00" containerID="f7417190086b0d6761a8e867854775b235d58f9ab5025f262e2cbb375df58f1f" exitCode=0 Jul 07 00:02:40.401825 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:40.401786 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" event={"ID":"07956dfa-fedf-47cb-8211-9768035b3c00","Type":"ContainerDied","Data":"f7417190086b0d6761a8e867854775b235d58f9ab5025f262e2cbb375df58f1f"} Jul 07 00:02:40.403760 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:40.403736 2563 generic.go:356] "Generic (PLEG): container finished" podID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerID="43c6ff659aea92e1a268fd2d1b3c8b18fbb052777150056ac2310b083b22765c" exitCode=0 Jul 07 00:02:40.403889 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:40.403823 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" event={"ID":"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376","Type":"ContainerDied","Data":"43c6ff659aea92e1a268fd2d1b3c8b18fbb052777150056ac2310b083b22765c"} Jul 07 00:02:40.406367 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:40.406345 2563 generic.go:356] "Generic (PLEG): container finished" podID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerID="f50b18bf999a870fc634ba1bd71de790b8d5e89b4e5011e404ef8d644b35b7ed" exitCode=0 Jul 07 00:02:40.406468 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:40.406410 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" event={"ID":"6136da15-a8e7-42a8-a387-1f4c7cd74ff7","Type":"ContainerDied","Data":"f50b18bf999a870fc634ba1bd71de790b8d5e89b4e5011e404ef8d644b35b7ed"} Jul 07 00:02:41.412792 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.412760 2563 generic.go:356] "Generic (PLEG): container finished" podID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerID="350d0e893df0e78d6e9c152f1280f579a6f3b32b1e462b0f1bd0a3fbda2fe8ae" exitCode=0 Jul 07 00:02:41.413235 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.412847 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" event={"ID":"50991f3d-a1cf-4c51-a611-1f9c136f7749","Type":"ContainerDied","Data":"350d0e893df0e78d6e9c152f1280f579a6f3b32b1e462b0f1bd0a3fbda2fe8ae"} Jul 07 00:02:41.415086 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.415058 2563 generic.go:356] "Generic (PLEG): container finished" podID="07956dfa-fedf-47cb-8211-9768035b3c00" containerID="ea9d225724c5b6b76ba8b5395d095e0b5b6075ca70c435e77178684279880a8c" exitCode=0 Jul 07 00:02:41.415234 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.415143 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" event={"ID":"07956dfa-fedf-47cb-8211-9768035b3c00","Type":"ContainerDied","Data":"ea9d225724c5b6b76ba8b5395d095e0b5b6075ca70c435e77178684279880a8c"} Jul 07 00:02:41.417197 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.417174 2563 generic.go:356] "Generic (PLEG): container finished" podID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerID="dac11b38a742424751ab2ba14f328fd58f2fc610feee81f2001db3664979c430" exitCode=0 Jul 07 00:02:41.417302 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.417202 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" event={"ID":"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376","Type":"ContainerDied","Data":"dac11b38a742424751ab2ba14f328fd58f2fc610feee81f2001db3664979c430"} Jul 07 00:02:41.553119 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.553092 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:41.667208 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.667116 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-bundle\") pod \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " Jul 07 00:02:41.667208 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.667164 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-util\") pod \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " Jul 07 00:02:41.667416 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.667213 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j969d\" (UniqueName: \"kubernetes.io/projected/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-kube-api-access-j969d\") pod \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\" (UID: \"6136da15-a8e7-42a8-a387-1f4c7cd74ff7\") " Jul 07 00:02:41.667656 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.667615 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-bundle" (OuterVolumeSpecName: "bundle") pod "6136da15-a8e7-42a8-a387-1f4c7cd74ff7" (UID: "6136da15-a8e7-42a8-a387-1f4c7cd74ff7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:41.669714 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.669688 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-kube-api-access-j969d" (OuterVolumeSpecName: "kube-api-access-j969d") pod "6136da15-a8e7-42a8-a387-1f4c7cd74ff7" (UID: "6136da15-a8e7-42a8-a387-1f4c7cd74ff7"). InnerVolumeSpecName "kube-api-access-j969d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:02:41.672404 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.672369 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-util" (OuterVolumeSpecName: "util") pod "6136da15-a8e7-42a8-a387-1f4c7cd74ff7" (UID: "6136da15-a8e7-42a8-a387-1f4c7cd74ff7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:41.768080 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.768041 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-util\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:41.768080 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.768077 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j969d\" (UniqueName: \"kubernetes.io/projected/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-kube-api-access-j969d\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:41.768080 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:41.768087 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6136da15-a8e7-42a8-a387-1f4c7cd74ff7-bundle\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:42.423672 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.423630 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" Jul 07 00:02:42.423672 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.423651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5" event={"ID":"6136da15-a8e7-42a8-a387-1f4c7cd74ff7","Type":"ContainerDied","Data":"fe3798cf25a9059270661861f2e987467473517ac5f1a60e1ae79246c09d9f8a"} Jul 07 00:02:42.424222 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.423736 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe3798cf25a9059270661861f2e987467473517ac5f1a60e1ae79246c09d9f8a" Jul 07 00:02:42.425777 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.425747 2563 generic.go:356] "Generic (PLEG): container finished" podID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerID="0afd15197fedf0ae4c5a48ac859d730564b8aaaf18b25468927d948b30697407" exitCode=0 Jul 07 00:02:42.425906 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.425779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" event={"ID":"50991f3d-a1cf-4c51-a611-1f9c136f7749","Type":"ContainerDied","Data":"0afd15197fedf0ae4c5a48ac859d730564b8aaaf18b25468927d948b30697407"} Jul 07 00:02:42.578884 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.578841 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:42.582031 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.582006 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:42.677227 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.677136 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-bundle\") pod \"07956dfa-fedf-47cb-8211-9768035b3c00\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " Jul 07 00:02:42.677227 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.677186 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2bm5\" (UniqueName: \"kubernetes.io/projected/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-kube-api-access-q2bm5\") pod \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " Jul 07 00:02:42.677227 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.677224 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-util\") pod \"07956dfa-fedf-47cb-8211-9768035b3c00\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " Jul 07 00:02:42.677477 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.677276 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prlbk\" (UniqueName: \"kubernetes.io/projected/07956dfa-fedf-47cb-8211-9768035b3c00-kube-api-access-prlbk\") pod \"07956dfa-fedf-47cb-8211-9768035b3c00\" (UID: \"07956dfa-fedf-47cb-8211-9768035b3c00\") " Jul 07 00:02:42.677477 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.677342 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-bundle\") pod \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " Jul 07 00:02:42.677477 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.677371 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-util\") pod \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\" (UID: \"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376\") " Jul 07 00:02:42.677875 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.677841 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-bundle" (OuterVolumeSpecName: "bundle") pod "07956dfa-fedf-47cb-8211-9768035b3c00" (UID: "07956dfa-fedf-47cb-8211-9768035b3c00"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:42.678174 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.677930 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-bundle" (OuterVolumeSpecName: "bundle") pod "ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" (UID: "ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:42.679570 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.679518 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-kube-api-access-q2bm5" (OuterVolumeSpecName: "kube-api-access-q2bm5") pod "ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" (UID: "ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376"). InnerVolumeSpecName "kube-api-access-q2bm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:02:42.679786 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.679765 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07956dfa-fedf-47cb-8211-9768035b3c00-kube-api-access-prlbk" (OuterVolumeSpecName: "kube-api-access-prlbk") pod "07956dfa-fedf-47cb-8211-9768035b3c00" (UID: "07956dfa-fedf-47cb-8211-9768035b3c00"). InnerVolumeSpecName "kube-api-access-prlbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:02:42.682286 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.682258 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-util" (OuterVolumeSpecName: "util") pod "07956dfa-fedf-47cb-8211-9768035b3c00" (UID: "07956dfa-fedf-47cb-8211-9768035b3c00"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:42.683255 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.683234 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-util" (OuterVolumeSpecName: "util") pod "ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" (UID: "ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:42.778673 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.778636 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-bundle\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:42.778673 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.778669 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bm5\" (UniqueName: \"kubernetes.io/projected/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-kube-api-access-q2bm5\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:42.778673 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.778680 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07956dfa-fedf-47cb-8211-9768035b3c00-util\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:42.778899 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.778688 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prlbk\" (UniqueName: \"kubernetes.io/projected/07956dfa-fedf-47cb-8211-9768035b3c00-kube-api-access-prlbk\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:42.778899 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.778697 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-bundle\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:42.778899 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:42.778704 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376-util\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:43.432374 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.432348 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" Jul 07 00:02:43.432843 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.432425 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs" event={"ID":"ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376","Type":"ContainerDied","Data":"058b2f980261520c7fdcc44ea2f7e6434c210872bf2052c893089e65f1824f25"} Jul 07 00:02:43.432843 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.432478 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="058b2f980261520c7fdcc44ea2f7e6434c210872bf2052c893089e65f1824f25" Jul 07 00:02:43.434412 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.434390 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" Jul 07 00:02:43.434589 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.434423 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k" event={"ID":"07956dfa-fedf-47cb-8211-9768035b3c00","Type":"ContainerDied","Data":"6d1b78757899524dad311d46496ea3a491269d1893167d6f8a163a38356b784f"} Jul 07 00:02:43.434589 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.434463 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d1b78757899524dad311d46496ea3a491269d1893167d6f8a163a38356b784f" Jul 07 00:02:43.461788 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:02:43.461749 2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9c918f_0b3e_45ef_a1c7_c6e39bbd0376.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07956dfa_fedf_47cb_8211_9768035b3c00.slice\": RecentStats: unable to find data in memory cache]" Jul 07 00:02:43.567578 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.567547 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:43.685183 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.685085 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-bundle\") pod \"50991f3d-a1cf-4c51-a611-1f9c136f7749\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " Jul 07 00:02:43.685183 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.685146 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-util\") pod \"50991f3d-a1cf-4c51-a611-1f9c136f7749\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " Jul 07 00:02:43.685404 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.685201 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz8c2\" (UniqueName: \"kubernetes.io/projected/50991f3d-a1cf-4c51-a611-1f9c136f7749-kube-api-access-xz8c2\") pod \"50991f3d-a1cf-4c51-a611-1f9c136f7749\" (UID: \"50991f3d-a1cf-4c51-a611-1f9c136f7749\") " Jul 07 00:02:43.685787 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.685758 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-bundle" (OuterVolumeSpecName: "bundle") pod "50991f3d-a1cf-4c51-a611-1f9c136f7749" (UID: "50991f3d-a1cf-4c51-a611-1f9c136f7749"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:43.687466 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.687441 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50991f3d-a1cf-4c51-a611-1f9c136f7749-kube-api-access-xz8c2" (OuterVolumeSpecName: "kube-api-access-xz8c2") pod "50991f3d-a1cf-4c51-a611-1f9c136f7749" (UID: "50991f3d-a1cf-4c51-a611-1f9c136f7749"). InnerVolumeSpecName "kube-api-access-xz8c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:02:43.690163 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.690123 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-util" (OuterVolumeSpecName: "util") pod "50991f3d-a1cf-4c51-a611-1f9c136f7749" (UID: "50991f3d-a1cf-4c51-a611-1f9c136f7749"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:43.786868 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.786820 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz8c2\" (UniqueName: \"kubernetes.io/projected/50991f3d-a1cf-4c51-a611-1f9c136f7749-kube-api-access-xz8c2\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:43.786868 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.786860 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-bundle\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:43.786868 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:43.786872 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50991f3d-a1cf-4c51-a611-1f9c136f7749-util\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:02:44.441346 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:44.441313 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" event={"ID":"50991f3d-a1cf-4c51-a611-1f9c136f7749","Type":"ContainerDied","Data":"98b34a524a54a3c26c9d70948d564855b8241085177045072ac3f3a17d39b59b"} Jul 07 00:02:44.441346 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:44.441343 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw" Jul 07 00:02:44.441858 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:44.441357 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98b34a524a54a3c26c9d70948d564855b8241085177045072ac3f3a17d39b59b" Jul 07 00:02:51.240900 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:51.240834 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:02:54.617610 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:54.617563 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:02:55.484163 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:55.484128 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:02:58.667956 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:58.667921 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:02:58.958876 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:02:58.958787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:00.981726 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:00.981682 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:02.746209 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:02.746175 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:04.278131 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:04.278092 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:05.489626 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:05.489583 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:07.426589 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:07.426515 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:07.526150 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:07.526114 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:08.837858 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:08.837831 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:08.939038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:08.939009 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:11.458198 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:11.458151 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:11.560070 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:11.560035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:11.867805 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:11.867726 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:12.060758 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.060717 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6"] Jul 07 00:03:12.061102 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061084 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerName="pull" Jul 07 00:03:12.061182 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061105 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerName="pull" Jul 07 00:03:12.061182 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061124 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerName="util" Jul 07 00:03:12.061182 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061132 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerName="util" Jul 07 00:03:12.061182 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061144 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerName="pull" Jul 07 00:03:12.061182 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061152 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerName="pull" Jul 07 00:03:12.061182 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061164 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerName="extract" Jul 07 00:03:12.061182 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061173 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061186 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07956dfa-fedf-47cb-8211-9768035b3c00" containerName="util" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061193 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07956dfa-fedf-47cb-8211-9768035b3c00" containerName="util" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061203 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerName="pull" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061214 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerName="pull" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061225 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerName="util" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061233 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerName="util" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061241 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07956dfa-fedf-47cb-8211-9768035b3c00" containerName="pull" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061248 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07956dfa-fedf-47cb-8211-9768035b3c00" containerName="pull" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061255 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061263 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061274 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07956dfa-fedf-47cb-8211-9768035b3c00" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061284 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07956dfa-fedf-47cb-8211-9768035b3c00" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061293 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061300 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061313 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerName="util" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061321 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerName="util" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061407 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="07956dfa-fedf-47cb-8211-9768035b3c00" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061419 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6136da15-a8e7-42a8-a387-1f4c7cd74ff7" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061432 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376" containerName="extract" Jul 07 00:03:12.061509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.061441 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50991f3d-a1cf-4c51-a611-1f9c136f7749" containerName="extract" Jul 07 00:03:12.064729 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.064708 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.068379 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.068348 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 00:03:12.068520 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.068388 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 00:03:12.068520 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.068478 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-htsf7\"" Jul 07 00:03:12.068872 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.068854 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 00:03:12.069143 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.069123 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 00:03:12.071251 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.071228 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6"] Jul 07 00:03:12.100915 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.100882 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:12.129762 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.129665 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb58690a-1cef-476c-923b-5ee24e057af7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.129762 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.129701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb58690a-1cef-476c-923b-5ee24e057af7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.129962 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.129802 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9crt\" (UniqueName: \"kubernetes.io/projected/fb58690a-1cef-476c-923b-5ee24e057af7-kube-api-access-p9crt\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.212078 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.212038 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:12.231193 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.231154 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb58690a-1cef-476c-923b-5ee24e057af7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.231193 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.231194 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb58690a-1cef-476c-923b-5ee24e057af7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.231393 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.231253 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9crt\" (UniqueName: \"kubernetes.io/projected/fb58690a-1cef-476c-923b-5ee24e057af7-kube-api-access-p9crt\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.231393 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:03:12.231314 2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 07 00:03:12.231461 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:03:12.231393 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb58690a-1cef-476c-923b-5ee24e057af7-plugin-serving-cert podName:fb58690a-1cef-476c-923b-5ee24e057af7 nodeName:}" failed. No retries permitted until 2026-07-07 00:03:12.731372106 +0000 UTC m=+430.676957474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/fb58690a-1cef-476c-923b-5ee24e057af7-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-dlpj6" (UID: "fb58690a-1cef-476c-923b-5ee24e057af7") : secret "plugin-serving-cert" not found Jul 07 00:03:12.231961 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.231944 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb58690a-1cef-476c-923b-5ee24e057af7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.243411 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.243385 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9crt\" (UniqueName: \"kubernetes.io/projected/fb58690a-1cef-476c-923b-5ee24e057af7-kube-api-access-p9crt\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.673915 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.673877 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:12.736384 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.736344 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb58690a-1cef-476c-923b-5ee24e057af7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.739102 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.739067 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb58690a-1cef-476c-923b-5ee24e057af7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dlpj6\" (UID: \"fb58690a-1cef-476c-923b-5ee24e057af7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:12.975416 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:12.975325 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" Jul 07 00:03:13.141771 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:13.141742 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6"] Jul 07 00:03:13.143862 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:03:13.143831 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb58690a_1cef_476c_923b_5ee24e057af7.slice/crio-27a342e2c6da64735935c11c1c21d01e091e6753fd5bab42d29117f473ca8ff9 WatchSource:0}: Error finding container 27a342e2c6da64735935c11c1c21d01e091e6753fd5bab42d29117f473ca8ff9: Status 404 returned error can't find the container with id 27a342e2c6da64735935c11c1c21d01e091e6753fd5bab42d29117f473ca8ff9 Jul 07 00:03:13.595937 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:13.595904 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" event={"ID":"fb58690a-1cef-476c-923b-5ee24e057af7","Type":"ContainerStarted","Data":"27a342e2c6da64735935c11c1c21d01e091e6753fd5bab42d29117f473ca8ff9"} Jul 07 00:03:14.875140 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:14.875047 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:15.029902 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:15.029859 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:19.586392 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:19.586351 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:19.692822 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:19.692787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:19.786224 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:19.786193 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:21.551971 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:21.551935 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:21.705105 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:21.705073 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:25.769109 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:25.769022 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:25.936741 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:25.936694 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:29.249372 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:29.249326 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:29.421158 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:29.421121 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:29.526810 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:29.526723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:33.408341 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:33.408304 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:33.520226 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:33.520191 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:34.730862 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:34.730821 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" event={"ID":"fb58690a-1cef-476c-923b-5ee24e057af7","Type":"ContainerStarted","Data":"d1fa70ddbcbcf01f1e1bd39ae2997e137569482cd4c1b77c7c0bcdc2df635285"} Jul 07 00:03:34.753403 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:34.753347 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dlpj6" podStartSLOduration=1.680793016 podStartE2EDuration="22.753330852s" totalImagesPullingTime="21.072537836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:03:34.751485044 +0000 UTC m=+452.697070415" watchObservedRunningTime="2026-07-07 00:03:34.753330852 +0000 UTC m=+452.698916223" Jul 07 00:03:39.974231 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:39.974195 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:40.082785 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:40.082749 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:53.392869 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:53.392816 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:53.832247 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:53.832212 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:55.678399 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:55.678362 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:55.748382 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:55.748354 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:55.820265 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:55.820225 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:55.971259 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:55.971184 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:03:56.020605 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.020560 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:03:56.020605 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.020607 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:03:56.020828 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.020635 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:03:56.023084 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.023060 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 00:03:56.032371 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.032347 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65nrf\" (UniqueName: \"kubernetes.io/projected/bf7d7d6f-c676-404c-8c19-cf1063d41a97-kube-api-access-65nrf\") pod \"limitador-limitador-54d6d988d5-nkt97\" (UID: \"bf7d7d6f-c676-404c-8c19-cf1063d41a97\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:03:56.032487 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.032404 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf7d7d6f-c676-404c-8c19-cf1063d41a97-config-file\") pod \"limitador-limitador-54d6d988d5-nkt97\" (UID: \"bf7d7d6f-c676-404c-8c19-cf1063d41a97\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:03:56.133037 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.133000 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65nrf\" (UniqueName: \"kubernetes.io/projected/bf7d7d6f-c676-404c-8c19-cf1063d41a97-kube-api-access-65nrf\") pod \"limitador-limitador-54d6d988d5-nkt97\" (UID: \"bf7d7d6f-c676-404c-8c19-cf1063d41a97\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:03:56.133213 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.133049 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf7d7d6f-c676-404c-8c19-cf1063d41a97-config-file\") pod \"limitador-limitador-54d6d988d5-nkt97\" (UID: \"bf7d7d6f-c676-404c-8c19-cf1063d41a97\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:03:56.133754 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.133736 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf7d7d6f-c676-404c-8c19-cf1063d41a97-config-file\") pod \"limitador-limitador-54d6d988d5-nkt97\" (UID: \"bf7d7d6f-c676-404c-8c19-cf1063d41a97\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:03:56.142921 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.142895 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65nrf\" (UniqueName: \"kubernetes.io/projected/bf7d7d6f-c676-404c-8c19-cf1063d41a97-kube-api-access-65nrf\") pod \"limitador-limitador-54d6d988d5-nkt97\" (UID: \"bf7d7d6f-c676-404c-8c19-cf1063d41a97\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:03:56.330937 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.330852 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:03:56.454522 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.454475 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-wb5cz"] Jul 07 00:03:56.491409 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.491376 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wb5cz"] Jul 07 00:03:56.491604 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.491492 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wb5cz" Jul 07 00:03:56.495342 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.495314 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zcg8l\"" Jul 07 00:03:56.499109 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.499085 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:03:56.502622 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:03:56.502587 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7d7d6f_c676_404c_8c19_cf1063d41a97.slice/crio-1c09593bf79000930a1e3730ebf30e9b2e1ea1ac8e6066cf478bc8df085f37f7 WatchSource:0}: Error finding container 1c09593bf79000930a1e3730ebf30e9b2e1ea1ac8e6066cf478bc8df085f37f7: Status 404 returned error can't find the container with id 1c09593bf79000930a1e3730ebf30e9b2e1ea1ac8e6066cf478bc8df085f37f7 Jul 07 00:03:56.503416 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.503396 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:56.536096 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.536051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zp4\" (UniqueName: \"kubernetes.io/projected/e3a13e26-33d6-448b-9422-ad42e7e21129-kube-api-access-t5zp4\") pod \"authorino-58cd955675-wb5cz\" (UID: \"e3a13e26-33d6-448b-9422-ad42e7e21129\") " pod="kuadrant-system/authorino-58cd955675-wb5cz" Jul 07 00:03:56.611594 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.611487 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:03:56.637347 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.637308 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zp4\" (UniqueName: \"kubernetes.io/projected/e3a13e26-33d6-448b-9422-ad42e7e21129-kube-api-access-t5zp4\") pod \"authorino-58cd955675-wb5cz\" (UID: \"e3a13e26-33d6-448b-9422-ad42e7e21129\") " pod="kuadrant-system/authorino-58cd955675-wb5cz" Jul 07 00:03:56.650587 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.650561 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zp4\" (UniqueName: \"kubernetes.io/projected/e3a13e26-33d6-448b-9422-ad42e7e21129-kube-api-access-t5zp4\") pod \"authorino-58cd955675-wb5cz\" (UID: \"e3a13e26-33d6-448b-9422-ad42e7e21129\") " pod="kuadrant-system/authorino-58cd955675-wb5cz" Jul 07 00:03:56.803548 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.803512 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wb5cz" Jul 07 00:03:56.844968 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.844927 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" event={"ID":"bf7d7d6f-c676-404c-8c19-cf1063d41a97","Type":"ContainerStarted","Data":"1c09593bf79000930a1e3730ebf30e9b2e1ea1ac8e6066cf478bc8df085f37f7"} Jul 07 00:03:56.971195 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:56.971156 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wb5cz"] Jul 07 00:03:56.973725 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:03:56.973690 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3a13e26_33d6_448b_9422_ad42e7e21129.slice/crio-d4cbd2ce5f95dd27d0e1817bfaa5a64984a3d2c2547defb57c81d7f105f59d9f WatchSource:0}: Error finding container d4cbd2ce5f95dd27d0e1817bfaa5a64984a3d2c2547defb57c81d7f105f59d9f: Status 404 returned error can't find the container with id d4cbd2ce5f95dd27d0e1817bfaa5a64984a3d2c2547defb57c81d7f105f59d9f Jul 07 00:03:57.854068 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:03:57.854010 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wb5cz" event={"ID":"e3a13e26-33d6-448b-9422-ad42e7e21129","Type":"ContainerStarted","Data":"d4cbd2ce5f95dd27d0e1817bfaa5a64984a3d2c2547defb57c81d7f105f59d9f"} Jul 07 00:04:00.875577 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:00.875510 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" event={"ID":"bf7d7d6f-c676-404c-8c19-cf1063d41a97","Type":"ContainerStarted","Data":"c4d6d6a056fce743811bd008c02b1a0d40dbe0170f0f1a94d88526829d008f96"} Jul 07 00:04:00.876155 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:00.875611 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:04:00.895402 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:00.895342 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" podStartSLOduration=2.59015779 podStartE2EDuration="5.895325687s" totalImagesPullingTime="3.305167897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:04:00.89360718 +0000 UTC m=+478.839192566" watchObservedRunningTime="2026-07-07 00:04:00.895325687 +0000 UTC m=+478.840911057" Jul 07 00:04:01.882094 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:01.882043 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wb5cz" event={"ID":"e3a13e26-33d6-448b-9422-ad42e7e21129","Type":"ContainerStarted","Data":"1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb"} Jul 07 00:04:01.907504 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:01.907450 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-wb5cz" podStartSLOduration=1.24355922 podStartE2EDuration="5.907434944s" totalImagesPullingTime="4.663875724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:04:01.906455193 +0000 UTC m=+479.852040565" watchObservedRunningTime="2026-07-07 00:04:01.907434944 +0000 UTC m=+479.853020315" Jul 07 00:04:02.090406 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:02.090371 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wb5cz"] Jul 07 00:04:03.891250 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:03.891198 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-wb5cz" podUID="e3a13e26-33d6-448b-9422-ad42e7e21129" containerName="authorino" containerID="cri-o://1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb" gracePeriod=30 Jul 07 00:04:04.154513 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.154482 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wb5cz" Jul 07 00:04:04.200862 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.200830 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5zp4\" (UniqueName: \"kubernetes.io/projected/e3a13e26-33d6-448b-9422-ad42e7e21129-kube-api-access-t5zp4\") pod \"e3a13e26-33d6-448b-9422-ad42e7e21129\" (UID: \"e3a13e26-33d6-448b-9422-ad42e7e21129\") " Jul 07 00:04:04.203043 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.203006 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a13e26-33d6-448b-9422-ad42e7e21129-kube-api-access-t5zp4" (OuterVolumeSpecName: "kube-api-access-t5zp4") pod "e3a13e26-33d6-448b-9422-ad42e7e21129" (UID: "e3a13e26-33d6-448b-9422-ad42e7e21129"). InnerVolumeSpecName "kube-api-access-t5zp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:04:04.301662 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.301623 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zp4\" (UniqueName: \"kubernetes.io/projected/e3a13e26-33d6-448b-9422-ad42e7e21129-kube-api-access-t5zp4\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:04:04.897812 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.897779 2563 generic.go:356] "Generic (PLEG): container finished" podID="e3a13e26-33d6-448b-9422-ad42e7e21129" containerID="1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb" exitCode=0 Jul 07 00:04:04.898224 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.897831 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wb5cz" Jul 07 00:04:04.898224 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.897864 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wb5cz" event={"ID":"e3a13e26-33d6-448b-9422-ad42e7e21129","Type":"ContainerDied","Data":"1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb"} Jul 07 00:04:04.898224 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.897906 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wb5cz" event={"ID":"e3a13e26-33d6-448b-9422-ad42e7e21129","Type":"ContainerDied","Data":"d4cbd2ce5f95dd27d0e1817bfaa5a64984a3d2c2547defb57c81d7f105f59d9f"} Jul 07 00:04:04.898224 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.897924 2563 scope.go:117] "RemoveContainer" containerID="1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb" Jul 07 00:04:04.906878 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.906862 2563 scope.go:117] "RemoveContainer" containerID="1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb" Jul 07 00:04:04.907195 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:04:04.907174 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb\": container with ID starting with 1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb not found: ID does not exist" containerID="1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb" Jul 07 00:04:04.907280 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.907202 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb"} err="failed to get container status \"1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb\": rpc error: code = NotFound desc = could not find container \"1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb\": container with ID starting with 1e6291e936bc8fa71da98489b714ab9f85e7532a8a4d1551fac9e00ef8dac5bb not found: ID does not exist" Jul 07 00:04:04.954908 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.954869 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wb5cz"] Jul 07 00:04:04.958548 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:04.958509 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-wb5cz"] Jul 07 00:04:06.576328 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:06.576296 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3a13e26-33d6-448b-9422-ad42e7e21129" path="/var/lib/kubelet/pods/e3a13e26-33d6-448b-9422-ad42e7e21129/volumes" Jul 07 00:04:09.878894 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:09.878846 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:11.883945 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:11.883916 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nkt97" Jul 07 00:04:17.365480 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:17.365442 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:20.339119 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:20.339079 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:29.495146 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.495099 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:29.725876 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.723333 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-x2c6g"] Jul 07 00:04:29.725876 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.724105 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a13e26-33d6-448b-9422-ad42e7e21129" containerName="authorino" Jul 07 00:04:29.725876 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.724126 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a13e26-33d6-448b-9422-ad42e7e21129" containerName="authorino" Jul 07 00:04:29.725876 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.724291 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a13e26-33d6-448b-9422-ad42e7e21129" containerName="authorino" Jul 07 00:04:29.727112 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.727085 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" Jul 07 00:04:29.730134 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.730108 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zcg8l\"" Jul 07 00:04:29.732123 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.732101 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-x2c6g"] Jul 07 00:04:29.823129 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.823029 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxnw\" (UniqueName: \"kubernetes.io/projected/c9c932ea-21b9-4ab7-8168-f62b27d9ae1e-kube-api-access-zcxnw\") pod \"authorino-5d4b67c446-x2c6g\" (UID: \"c9c932ea-21b9-4ab7-8168-f62b27d9ae1e\") " pod="kuadrant-system/authorino-5d4b67c446-x2c6g" Jul 07 00:04:29.907166 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.907138 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:29.924422 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.924393 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxnw\" (UniqueName: \"kubernetes.io/projected/c9c932ea-21b9-4ab7-8168-f62b27d9ae1e-kube-api-access-zcxnw\") pod \"authorino-5d4b67c446-x2c6g\" (UID: \"c9c932ea-21b9-4ab7-8168-f62b27d9ae1e\") " pod="kuadrant-system/authorino-5d4b67c446-x2c6g" Jul 07 00:04:29.933044 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.933007 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxnw\" (UniqueName: \"kubernetes.io/projected/c9c932ea-21b9-4ab7-8168-f62b27d9ae1e-kube-api-access-zcxnw\") pod \"authorino-5d4b67c446-x2c6g\" (UID: \"c9c932ea-21b9-4ab7-8168-f62b27d9ae1e\") " pod="kuadrant-system/authorino-5d4b67c446-x2c6g" Jul 07 00:04:29.959937 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.959901 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-x2c6g"] Jul 07 00:04:29.960167 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.960155 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" Jul 07 00:04:29.988402 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.988372 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64b6bc7698-pxbq9"] Jul 07 00:04:29.990832 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.990810 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" Jul 07 00:04:29.999673 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:29.999645 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64b6bc7698-pxbq9"] Jul 07 00:04:30.100858 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:30.100775 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-x2c6g"] Jul 07 00:04:30.104214 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:04:30.104184 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9c932ea_21b9_4ab7_8168_f62b27d9ae1e.slice/crio-96db2b5392696da7e1a8150354e9908f5353176c6726df829662b7466105566c WatchSource:0}: Error finding container 96db2b5392696da7e1a8150354e9908f5353176c6726df829662b7466105566c: Status 404 returned error can't find the container with id 96db2b5392696da7e1a8150354e9908f5353176c6726df829662b7466105566c Jul 07 00:04:30.126142 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:30.126113 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpf27\" (UniqueName: \"kubernetes.io/projected/b814e371-16db-4e82-a247-61814a3ca6c1-kube-api-access-lpf27\") pod \"authorino-64b6bc7698-pxbq9\" (UID: \"b814e371-16db-4e82-a247-61814a3ca6c1\") " pod="kuadrant-system/authorino-64b6bc7698-pxbq9" Jul 07 00:04:30.227332 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:30.227293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpf27\" (UniqueName: \"kubernetes.io/projected/b814e371-16db-4e82-a247-61814a3ca6c1-kube-api-access-lpf27\") pod \"authorino-64b6bc7698-pxbq9\" (UID: \"b814e371-16db-4e82-a247-61814a3ca6c1\") " pod="kuadrant-system/authorino-64b6bc7698-pxbq9" Jul 07 00:04:30.238148 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:30.238112 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpf27\" (UniqueName: \"kubernetes.io/projected/b814e371-16db-4e82-a247-61814a3ca6c1-kube-api-access-lpf27\") pod \"authorino-64b6bc7698-pxbq9\" (UID: \"b814e371-16db-4e82-a247-61814a3ca6c1\") " pod="kuadrant-system/authorino-64b6bc7698-pxbq9" Jul 07 00:04:30.241962 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:30.241933 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64b6bc7698-pxbq9"] Jul 07 00:04:30.242197 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:30.242184 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" Jul 07 00:04:30.374650 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:30.374621 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64b6bc7698-pxbq9"] Jul 07 00:04:30.376433 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:04:30.376411 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb814e371_16db_4e82_a247_61814a3ca6c1.slice/crio-ea6b36afce5ae4a5150a27b9b22e743d937e1004e3e200c8837767686ce6e72b WatchSource:0}: Error finding container ea6b36afce5ae4a5150a27b9b22e743d937e1004e3e200c8837767686ce6e72b: Status 404 returned error can't find the container with id ea6b36afce5ae4a5150a27b9b22e743d937e1004e3e200c8837767686ce6e72b Jul 07 00:04:31.029101 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.029069 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" event={"ID":"b814e371-16db-4e82-a247-61814a3ca6c1","Type":"ContainerStarted","Data":"e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1"} Jul 07 00:04:31.029101 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.029109 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" event={"ID":"b814e371-16db-4e82-a247-61814a3ca6c1","Type":"ContainerStarted","Data":"ea6b36afce5ae4a5150a27b9b22e743d937e1004e3e200c8837767686ce6e72b"} Jul 07 00:04:31.029645 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.029127 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" podUID="b814e371-16db-4e82-a247-61814a3ca6c1" containerName="authorino" containerID="cri-o://e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1" gracePeriod=30 Jul 07 00:04:31.030901 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.030875 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" podUID="c9c932ea-21b9-4ab7-8168-f62b27d9ae1e" containerName="authorino" containerID="cri-o://394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7" gracePeriod=30 Jul 07 00:04:31.030998 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.030878 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" event={"ID":"c9c932ea-21b9-4ab7-8168-f62b27d9ae1e","Type":"ContainerStarted","Data":"394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7"} Jul 07 00:04:31.030998 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.030988 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" event={"ID":"c9c932ea-21b9-4ab7-8168-f62b27d9ae1e","Type":"ContainerStarted","Data":"96db2b5392696da7e1a8150354e9908f5353176c6726df829662b7466105566c"} Jul 07 00:04:31.046766 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.046711 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" podStartSLOduration=1.720804548 podStartE2EDuration="2.046688917s" totalImagesPullingTime="325.884369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:04:31.044220516 +0000 UTC m=+508.989805888" watchObservedRunningTime="2026-07-07 00:04:31.046688917 +0000 UTC m=+508.992274290" Jul 07 00:04:31.063565 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.063496 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" podStartSLOduration=1.547541764 podStartE2EDuration="2.063480156s" totalImagesPullingTime="515.938392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:04:31.062161142 +0000 UTC m=+509.007746515" watchObservedRunningTime="2026-07-07 00:04:31.063480156 +0000 UTC m=+509.009065534" Jul 07 00:04:31.300855 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.300828 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" Jul 07 00:04:31.321285 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.321257 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" Jul 07 00:04:31.444579 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.444516 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcxnw\" (UniqueName: \"kubernetes.io/projected/c9c932ea-21b9-4ab7-8168-f62b27d9ae1e-kube-api-access-zcxnw\") pod \"c9c932ea-21b9-4ab7-8168-f62b27d9ae1e\" (UID: \"c9c932ea-21b9-4ab7-8168-f62b27d9ae1e\") " Jul 07 00:04:31.444766 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.444612 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpf27\" (UniqueName: \"kubernetes.io/projected/b814e371-16db-4e82-a247-61814a3ca6c1-kube-api-access-lpf27\") pod \"b814e371-16db-4e82-a247-61814a3ca6c1\" (UID: \"b814e371-16db-4e82-a247-61814a3ca6c1\") " Jul 07 00:04:31.446753 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.446724 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b814e371-16db-4e82-a247-61814a3ca6c1-kube-api-access-lpf27" (OuterVolumeSpecName: "kube-api-access-lpf27") pod "b814e371-16db-4e82-a247-61814a3ca6c1" (UID: "b814e371-16db-4e82-a247-61814a3ca6c1"). InnerVolumeSpecName "kube-api-access-lpf27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:04:31.446859 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.446764 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c932ea-21b9-4ab7-8168-f62b27d9ae1e-kube-api-access-zcxnw" (OuterVolumeSpecName: "kube-api-access-zcxnw") pod "c9c932ea-21b9-4ab7-8168-f62b27d9ae1e" (UID: "c9c932ea-21b9-4ab7-8168-f62b27d9ae1e"). InnerVolumeSpecName "kube-api-access-zcxnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:04:31.546006 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.545915 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcxnw\" (UniqueName: \"kubernetes.io/projected/c9c932ea-21b9-4ab7-8168-f62b27d9ae1e-kube-api-access-zcxnw\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:04:31.546006 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:31.545948 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpf27\" (UniqueName: \"kubernetes.io/projected/b814e371-16db-4e82-a247-61814a3ca6c1-kube-api-access-lpf27\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:04:32.037340 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.037300 2563 generic.go:356] "Generic (PLEG): container finished" podID="b814e371-16db-4e82-a247-61814a3ca6c1" containerID="e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1" exitCode=0 Jul 07 00:04:32.037340 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.037342 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" Jul 07 00:04:32.037908 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.037390 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" event={"ID":"b814e371-16db-4e82-a247-61814a3ca6c1","Type":"ContainerDied","Data":"e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1"} Jul 07 00:04:32.037908 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.037425 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b6bc7698-pxbq9" event={"ID":"b814e371-16db-4e82-a247-61814a3ca6c1","Type":"ContainerDied","Data":"ea6b36afce5ae4a5150a27b9b22e743d937e1004e3e200c8837767686ce6e72b"} Jul 07 00:04:32.037908 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.037448 2563 scope.go:117] "RemoveContainer" containerID="e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1" Jul 07 00:04:32.038791 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.038733 2563 generic.go:356] "Generic (PLEG): container finished" podID="c9c932ea-21b9-4ab7-8168-f62b27d9ae1e" containerID="394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7" exitCode=0 Jul 07 00:04:32.038791 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.038769 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" Jul 07 00:04:32.038921 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.038834 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" event={"ID":"c9c932ea-21b9-4ab7-8168-f62b27d9ae1e","Type":"ContainerDied","Data":"394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7"} Jul 07 00:04:32.038921 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.038860 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-x2c6g" event={"ID":"c9c932ea-21b9-4ab7-8168-f62b27d9ae1e","Type":"ContainerDied","Data":"96db2b5392696da7e1a8150354e9908f5353176c6726df829662b7466105566c"} Jul 07 00:04:32.047211 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.047191 2563 scope.go:117] "RemoveContainer" containerID="e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1" Jul 07 00:04:32.047549 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:04:32.047509 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1\": container with ID starting with e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1 not found: ID does not exist" containerID="e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1" Jul 07 00:04:32.047659 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.047565 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1"} err="failed to get container status \"e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1\": rpc error: code = NotFound desc = could not find container \"e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1\": container with ID starting with e78c92bebf45d3acd9bc8c0f995349e7cb52a31f4f4bf56cfb8b499c318f57d1 not found: ID does not exist" Jul 07 00:04:32.047659 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.047583 2563 scope.go:117] "RemoveContainer" containerID="394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7" Jul 07 00:04:32.056448 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.056427 2563 scope.go:117] "RemoveContainer" containerID="394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7" Jul 07 00:04:32.056749 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:04:32.056729 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7\": container with ID starting with 394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7 not found: ID does not exist" containerID="394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7" Jul 07 00:04:32.056809 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.056757 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7"} err="failed to get container status \"394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7\": rpc error: code = NotFound desc = could not find container \"394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7\": container with ID starting with 394e6d11c4559c5e8607224056635dd80918587b125ec4eb19320ea4017edbb7 not found: ID does not exist" Jul 07 00:04:32.066715 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.066684 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64b6bc7698-pxbq9"] Jul 07 00:04:32.073606 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.073579 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64b6bc7698-pxbq9"] Jul 07 00:04:32.090136 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.090100 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-x2c6g"] Jul 07 00:04:32.095699 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.095676 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-x2c6g"] Jul 07 00:04:32.576375 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.576339 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b814e371-16db-4e82-a247-61814a3ca6c1" path="/var/lib/kubelet/pods/b814e371-16db-4e82-a247-61814a3ca6c1/volumes" Jul 07 00:04:32.576785 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:32.576766 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c932ea-21b9-4ab7-8168-f62b27d9ae1e" path="/var/lib/kubelet/pods/c9c932ea-21b9-4ab7-8168-f62b27d9ae1e/volumes" Jul 07 00:04:33.813983 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:33.813953 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:33.914490 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:33.914460 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:36.555162 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:36.555129 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:36.655164 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:36.655122 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:36.745823 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:36.745788 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:36.807976 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:36.807893 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:39.712550 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:39.712505 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:39.834527 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:39.834490 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:04:57.259343 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:04:57.259310 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:12.073420 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:12.073369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:12.516466 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:12.516434 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:12.849888 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:12.849800 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:13.180493 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:13.180456 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:05:14.326408 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:14.326370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:14.426894 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:14.426858 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:20.133251 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:20.133213 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:20.194814 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:20.194772 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:22.952993 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:22.952938 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:23.053049 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:23.053010 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:30.593731 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:30.593696 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:31.235649 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:31.235613 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:05:37.914256 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:37.914200 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:05:38.645664 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:38.645629 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:05:42.427709 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:05:42.427675 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:06:01.363063 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.363027 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq"] Jul 07 00:06:01.363512 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.363336 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c932ea-21b9-4ab7-8168-f62b27d9ae1e" containerName="authorino" Jul 07 00:06:01.363512 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.363346 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c932ea-21b9-4ab7-8168-f62b27d9ae1e" containerName="authorino" Jul 07 00:06:01.363512 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.363363 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b814e371-16db-4e82-a247-61814a3ca6c1" containerName="authorino" Jul 07 00:06:01.363512 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.363368 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b814e371-16db-4e82-a247-61814a3ca6c1" containerName="authorino" Jul 07 00:06:01.363512 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.363423 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b814e371-16db-4e82-a247-61814a3ca6c1" containerName="authorino" Jul 07 00:06:01.363512 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.363432 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c932ea-21b9-4ab7-8168-f62b27d9ae1e" containerName="authorino" Jul 07 00:06:01.365469 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.365451 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.371058 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.371033 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 00:06:01.371058 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.371049 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 07 00:06:01.371330 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.371058 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 00:06:01.371330 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.371100 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-st4mb\"" Jul 07 00:06:01.377793 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.377767 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq"] Jul 07 00:06:01.396839 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.396800 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.396839 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.396839 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78466abb-2729-4d5f-8501-a53f11132356-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.397033 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.396865 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.397033 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.396964 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.397033 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.397010 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.397130 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.397087 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvxs\" (UniqueName: \"kubernetes.io/projected/78466abb-2729-4d5f-8501-a53f11132356-kube-api-access-gcvxs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.398328 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.398304 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:06:01.497680 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.497637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvxs\" (UniqueName: \"kubernetes.io/projected/78466abb-2729-4d5f-8501-a53f11132356-kube-api-access-gcvxs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.497870 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.497690 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.497870 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.497715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78466abb-2729-4d5f-8501-a53f11132356-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.497870 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.497744 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.497870 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.497790 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.497870 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.497814 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.498140 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.498074 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.498140 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.498116 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.498415 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.498391 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.500187 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.500162 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78466abb-2729-4d5f-8501-a53f11132356-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.500969 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.500951 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78466abb-2729-4d5f-8501-a53f11132356-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.506770 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.506749 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvxs\" (UniqueName: \"kubernetes.io/projected/78466abb-2729-4d5f-8501-a53f11132356-kube-api-access-gcvxs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-54xwq\" (UID: \"78466abb-2729-4d5f-8501-a53f11132356\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.676385 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.676347 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:01.815062 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:01.814958 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq"] Jul 07 00:06:01.817438 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:06:01.817402 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78466abb_2729_4d5f_8501_a53f11132356.slice/crio-b89b3f27e3abe895a787049cb6707af03ef9b5d5cb2759180bc28dee0216801d WatchSource:0}: Error finding container b89b3f27e3abe895a787049cb6707af03ef9b5d5cb2759180bc28dee0216801d: Status 404 returned error can't find the container with id b89b3f27e3abe895a787049cb6707af03ef9b5d5cb2759180bc28dee0216801d Jul 07 00:06:02.348437 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:02.348402 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:06:02.480510 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:02.480462 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" event={"ID":"78466abb-2729-4d5f-8501-a53f11132356","Type":"ContainerStarted","Data":"b89b3f27e3abe895a787049cb6707af03ef9b5d5cb2759180bc28dee0216801d"} Jul 07 00:06:03.487138 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:03.487098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" event={"ID":"78466abb-2729-4d5f-8501-a53f11132356","Type":"ContainerStarted","Data":"df809885b31df541b1e3ef57bafab9fc6e5d3ff9d18256d75a9cab226fb8f801"} Jul 07 00:06:03.508925 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:03.508872 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" podStartSLOduration=1.480546291 podStartE2EDuration="2.508856404s" totalImagesPullingTime="1.028310113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:06:03.505921229 +0000 UTC m=+601.451506601" watchObservedRunningTime="2026-07-07 00:06:03.508856404 +0000 UTC m=+601.454441827" Jul 07 00:06:05.205701 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:05.205662 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:06:07.352760 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:07.352727 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:06:11.676648 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:11.676608 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:11.677033 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:11.676678 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:11.689344 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:11.689318 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:12.543767 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:12.543736 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-54xwq" Jul 07 00:06:16.559996 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.559955 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg"] Jul 07 00:06:16.563031 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.563011 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.565601 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.565577 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 07 00:06:16.577407 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.577375 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg"] Jul 07 00:06:16.588086 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.588056 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:06:16.628246 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.628206 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.628427 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.628329 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.628427 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.628361 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dea6624-519f-4aad-872d-9cb50ed978ff-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.628583 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.628487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.628583 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.628575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.628698 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.628606 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfcft\" (UniqueName: \"kubernetes.io/projected/2dea6624-519f-4aad-872d-9cb50ed978ff-kube-api-access-rfcft\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.729635 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.729585 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.729922 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.729668 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.729922 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.729687 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dea6624-519f-4aad-872d-9cb50ed978ff-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.729922 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.729745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.729922 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.729770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.729922 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.729785 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfcft\" (UniqueName: \"kubernetes.io/projected/2dea6624-519f-4aad-872d-9cb50ed978ff-kube-api-access-rfcft\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.730148 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.729961 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.730148 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.729985 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.730148 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.730111 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.732238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.732215 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dea6624-519f-4aad-872d-9cb50ed978ff-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.732394 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.732376 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dea6624-519f-4aad-872d-9cb50ed978ff-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.738321 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.738291 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfcft\" (UniqueName: \"kubernetes.io/projected/2dea6624-519f-4aad-872d-9cb50ed978ff-kube-api-access-rfcft\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2r4rg\" (UID: \"2dea6624-519f-4aad-872d-9cb50ed978ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:16.874121 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:16.874020 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:17.013656 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:17.013604 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg"] Jul 07 00:06:17.016030 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:06:17.015996 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dea6624_519f_4aad_872d_9cb50ed978ff.slice/crio-d9325d0078c525e14fd639233676ad58c8f5a24877106a972c1d2d06f8355167 WatchSource:0}: Error finding container d9325d0078c525e14fd639233676ad58c8f5a24877106a972c1d2d06f8355167: Status 404 returned error can't find the container with id d9325d0078c525e14fd639233676ad58c8f5a24877106a972c1d2d06f8355167 Jul 07 00:06:17.017488 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:17.017467 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 00:06:17.559846 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:17.559758 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" event={"ID":"2dea6624-519f-4aad-872d-9cb50ed978ff","Type":"ContainerStarted","Data":"902f4a1c79b4378b36ab2de93604e3590881cbefaa7409971a329741b8739062"} Jul 07 00:06:17.559846 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:17.559812 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" event={"ID":"2dea6624-519f-4aad-872d-9cb50ed978ff","Type":"ContainerStarted","Data":"d9325d0078c525e14fd639233676ad58c8f5a24877106a972c1d2d06f8355167"} Jul 07 00:06:17.580999 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:17.580948 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" podStartSLOduration=1.383403114 podStartE2EDuration="1.580929165s" totalImagesPullingTime="197.526051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:06:17.578908373 +0000 UTC m=+615.524493758" watchObservedRunningTime="2026-07-07 00:06:17.580929165 +0000 UTC m=+615.526514615" Jul 07 00:06:18.245318 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:18.245271 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:06:26.239765 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.239667 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8"] Jul 07 00:06:26.242688 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.242662 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.245763 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.245737 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 00:06:26.253800 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.253772 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8"] Jul 07 00:06:26.290141 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.290107 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:06:26.316184 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.316140 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.316369 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.316192 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.316369 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.316254 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.316369 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.316339 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.316369 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.316364 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.316509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.316410 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fkdz\" (UniqueName: \"kubernetes.io/projected/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-kube-api-access-5fkdz\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.417022 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.416984 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.417022 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.417023 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.417288 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.417050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.417288 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.417099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.417288 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.417114 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.417288 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.417158 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fkdz\" (UniqueName: \"kubernetes.io/projected/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-kube-api-access-5fkdz\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.417496 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.417442 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.417496 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.417466 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.417677 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.417613 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.419631 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.419608 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.419865 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.419845 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.426128 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.426107 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fkdz\" (UniqueName: \"kubernetes.io/projected/8cdf799b-ea6f-45ee-bacb-bc86e019ddee-kube-api-access-5fkdz\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8\" (UID: \"8cdf799b-ea6f-45ee-bacb-bc86e019ddee\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.554685 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.554587 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:26.695518 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.695491 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8"] Jul 07 00:06:26.697939 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:06:26.697911 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cdf799b_ea6f_45ee_bacb_bc86e019ddee.slice/crio-e929beb967e7a7ef380a1e1af61b501d9a06df397eb2e1db33d09aa0c4ed0ffe WatchSource:0}: Error finding container e929beb967e7a7ef380a1e1af61b501d9a06df397eb2e1db33d09aa0c4ed0ffe: Status 404 returned error can't find the container with id e929beb967e7a7ef380a1e1af61b501d9a06df397eb2e1db33d09aa0c4ed0ffe Jul 07 00:06:26.874730 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.874641 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:26.874730 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.874701 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:26.887557 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:26.887504 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:27.613765 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:27.613729 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" event={"ID":"8cdf799b-ea6f-45ee-bacb-bc86e019ddee","Type":"ContainerStarted","Data":"8a1ff6fd41cee45e6f23d3e72405e15a4e7f3e5607055886cfa9750cf19c05ad"} Jul 07 00:06:27.613765 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:27.613782 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" event={"ID":"8cdf799b-ea6f-45ee-bacb-bc86e019ddee","Type":"ContainerStarted","Data":"e929beb967e7a7ef380a1e1af61b501d9a06df397eb2e1db33d09aa0c4ed0ffe"} Jul 07 00:06:27.625213 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:27.625185 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2r4rg" Jul 07 00:06:27.632187 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:27.632144 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" podStartSLOduration=1.440973394 podStartE2EDuration="1.632128389s" totalImagesPullingTime="191.154995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:06:27.630744316 +0000 UTC m=+625.576329700" watchObservedRunningTime="2026-07-07 00:06:27.632128389 +0000 UTC m=+625.577713760" Jul 07 00:06:28.748141 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:28.748103 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:06:36.555739 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:36.555702 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:36.556195 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:36.555980 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:36.568833 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:36.568803 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:06:36.673933 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:06:36.673903 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8" Jul 07 00:07:13.847705 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:07:13.847669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:13.946430 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:07:13.946394 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:14.961906 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:07:14.961870 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:15.059083 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:07:15.059054 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:07:42.650483 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:07:42.650439 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:07:51.942332 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:07:51.942293 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:08:03.442157 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:08:03.442117 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:08:12.147341 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:08:12.147308 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:08:23.348034 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:08:23.347999 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:08:25.403888 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:08:25.403858 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:26.821090 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:08:26.821053 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:08:51.845474 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:08:51.845444 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:09:02.545848 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:02.545813 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:09:10.541727 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:10.541688 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:10.637833 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:10.637801 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:10.869509 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:10.869424 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:16.867900 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:16.867862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:17.113260 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:17.113217 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:17.218749 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:17.218712 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:17.479042 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:17.478956 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:23.050072 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:23.049982 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:23.146787 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:23.146750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:09:39.543906 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:39.543867 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:09:47.853982 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:47.853944 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:09:56.948141 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:09:56.948106 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:10:02.748943 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:10:02.748905 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:10:13.855874 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:10:13.855837 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:10:23.045547 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:10:23.045497 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:10:34.337399 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:10:34.337349 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:10:43.745148 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:10:43.745113 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:11:01.703087 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:11:01.703053 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:02.971317 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:11:02.971278 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:06.841050 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:11:06.841014 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:11:18.042256 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:11:18.042211 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:11:25.717835 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:11:25.717803 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:11:42.241192 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:11:42.241154 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:11:53.039931 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:11:53.039892 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:12:06.956173 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:12:06.956127 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:12:16.444616 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:12:16.444579 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:12:25.449850 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:12:25.449766 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:12:32.317819 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:12:32.317774 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:32.415501 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:12:32.415460 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:12:35.044372 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:12:35.044335 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:12:43.649061 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:12:43.649016 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:14:35.852360 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:14:35.852325 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:14:44.445681 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:14:44.445645 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:14:58.940636 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:14:58.940595 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:15:09.048354 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:15:09.048296 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:15:19.784714 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:15:19.784677 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:15:31.741251 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:15:31.741213 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:15:39.547625 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:15:39.547588 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:15:50.741211 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:15:50.741165 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:16:02.143014 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:16:02.142982 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:16:10.844341 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:16:10.844302 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:16:23.039373 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:16:23.039320 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:16:43.842026 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:16:43.841978 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:16:50.950584 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:16:50.950524 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:17:00.440192 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:17:00.440102 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:17:00.583107 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:17:00.583065 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:17:10.643301 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:17:10.643261 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:17:16.949102 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:17:16.949061 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:17:30.753963 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:17:30.753924 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:17:42.444838 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:17:42.444802 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:17:51.543601 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:17:51.543551 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:18:01.147571 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:18:01.147518 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:18:09.644725 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:18:09.644688 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:18:18.348001 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:18:18.347962 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:18:32.045710 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:18:32.045625 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:18:45.347684 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:18:45.347644 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:18:58.147877 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:18:58.147837 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:18:58.825683 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:18:58.825651 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:19:07.645549 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:19:07.645486 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:19:17.448834 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:19:17.448775 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:19:34.348042 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:19:34.348000 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:19:42.673717 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:19:42.673685 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:19:45.554049 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:19:45.554011 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:19:54.645522 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:19:54.645440 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:19:57.595773 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:19:57.595739 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:05.159700 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:05.159656 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:07.844448 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:07.844408 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:20:14.225848 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:14.225805 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:18.141403 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:18.141360 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:20:18.400036 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:18.400001 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:18.620609 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:18.620577 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:21.138920 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:21.138887 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:21.237672 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:21.237642 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:21.580718 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:21.580688 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:27.396697 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:27.396653 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:27.496866 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:27.496828 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:27.591641 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:27.591606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:27.650916 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:27.650821 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:20:27.661565 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:27.661514 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:28.797845 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:28.797786 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:28.899153 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:28.899121 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:28.971447 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:28.971415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:29.163052 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:29.163023 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:29.447275 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:29.447198 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:29.547267 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:29.547231 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:31.088589 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:31.088556 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:33.793802 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:33.793765 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:34.006491 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:34.006454 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:34.107737 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:34.107658 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:34.359701 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:34.359614 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:35.466406 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:35.466368 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:39.712855 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:39.712822 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:39.771737 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:39.771699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:39.926292 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:39.926248 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:40.022871 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:40.022778 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:41.020698 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:41.020661 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:41.101029 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:41.100985 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:41.174636 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:41.174606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:41.264974 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:41.264937 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:41.680487 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:41.680453 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:41.790353 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:41.790320 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:43.277137 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:43.277101 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:43.448908 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:43.448869 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:20:46.138001 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:46.137969 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:46.693362 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:46.693315 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:52.215174 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:52.215137 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:52.325808 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:52.325760 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:54.541137 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:54.541099 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:20:58.233169 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:58.233134 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:58.341222 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:58.341191 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:58.454272 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:58.454236 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:59.186896 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:59.186859 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:59.287596 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:59.287550 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:59.654984 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:59.654948 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:59.733088 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:59.733059 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:20:59.799745 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:20:59.799708 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:00.204195 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:00.204166 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:00.304186 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:00.304148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:00.404146 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:00.404112 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:02.793469 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:02.793423 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:04.022328 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:04.022294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:04.215301 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:04.215268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:04.320464 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:04.320352 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:04.425115 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:04.425077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:05.587924 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:05.587887 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:10.254403 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:10.254368 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:10.312964 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:10.312912 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:10.760147 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:10.760103 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:10.843321 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:10.843291 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:10.942735 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:10.942686 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:12.347587 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:12.347557 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:18.195276 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:18.195238 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:18.298894 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:18.298862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:18.393938 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:18.393899 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:18.494169 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:18.494091 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:18.965011 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:18.964975 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:44.944505 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:44.944470 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:50.556418 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:50.556384 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:50.618264 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:50.618220 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:50.713677 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:50.713647 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:21:50.835264 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:21:50.835173 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:06.710233 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:06.710200 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:12.869247 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:12.869210 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:12.929181 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:12.929147 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:13.012781 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:13.012746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:13.108291 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:13.108262 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:23.600560 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:23.600496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:24.428153 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:24.428107 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:30.041191 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:30.041160 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:30.153095 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:30.153059 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:30.246931 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:30.246898 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:30.344227 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:30.344150 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:31.147019 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:31.146986 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:31.238712 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:31.238678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:31.337820 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:31.337795 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:32.249799 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:32.249766 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:22:32.934396 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:32.934355 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:38.842002 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:38.841964 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:38.942679 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:38.942638 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:39.449759 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:39.449729 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:39.508599 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:39.508561 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:22:45.416898 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:45.416858 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:45.517654 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:45.517615 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:45.616975 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:45.616934 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:45.946481 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:45.946435 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:46.119315 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:46.119284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:47.847212 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:47.847163 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:22:48.833411 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:48.833381 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:48.925215 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:48.925177 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:49.120101 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:49.120012 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:49.471950 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:49.471907 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:49.713548 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:49.713493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:49.814112 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:49.814032 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:53.329309 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:53.329219 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:53.986271 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:53.986228 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:54.099857 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:54.099824 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:54.426486 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:54.426441 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:22:57.456640 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:22:57.456601 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:23:00.458689 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:00.458639 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:00.525599 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:00.525559 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:00.862793 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:00.862684 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:01.248243 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:01.248212 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:01.349263 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:01.349220 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:04.345083 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:04.345046 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:23:07.063276 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:07.063230 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:07.164873 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:07.164835 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:07.340823 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:07.340733 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:07.548901 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:07.548856 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:07.690213 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:07.690182 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:07.784210 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:07.784154 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:07.899372 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:07.899332 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:13.556806 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:13.556766 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:13.667764 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:13.667715 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:13.758372 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:13.758338 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:13.891587 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:13.891470 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:19.877688 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:19.877655 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:19.958494 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:19.958460 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:26.695322 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:26.695290 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:27.009034 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:27.008959 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:27.109356 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:27.109322 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:29.991456 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:29.991424 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:30.091943 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:30.091910 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:30.278238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:30.278151 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:30.410656 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:30.410619 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:30.767079 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:30.767041 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:30.869735 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:30.869702 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:34.481066 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:34.481032 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:35.242830 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:35.242792 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:23:41.752898 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.752857 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6"] Jul 07 00:23:41.756390 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.756353 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.757128 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.757104 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tls-certs\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.757253 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.757137 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g47cz\" (UniqueName: \"kubernetes.io/projected/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-kube-api-access-g47cz\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.757253 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.757185 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-dshm\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.757253 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.757233 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tmp-dir\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.757376 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.757277 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-home\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.757376 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.757295 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-model-cache\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.759105 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.759087 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7d49bf96\"/\"openshift-service-ca.crt\"" Jul 07 00:23:41.759328 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.759309 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7d49bf96\"/\"kube-root-ca.crt\"" Jul 07 00:23:41.760417 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.760397 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7d49bf96\"/\"e2e-lifecycle-model-7d49bf96-kserve-self-signed-certs\"" Jul 07 00:23:41.760517 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.760466 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7d49bf96\"/\"default-dockercfg-747jj\"" Jul 07 00:23:41.772191 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.772165 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6"] Jul 07 00:23:41.845513 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.845481 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:41.858385 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.858350 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-dshm\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.858385 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.858384 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tmp-dir\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.858628 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.858412 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-home\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.858628 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.858477 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-model-cache\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.858628 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.858568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tls-certs\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.858628 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.858598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g47cz\" (UniqueName: \"kubernetes.io/projected/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-kube-api-access-g47cz\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.858803 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.858784 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-home\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.858890 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.858872 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tmp-dir\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.858945 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.858901 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-model-cache\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.860679 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.860658 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-dshm\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.861143 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.861124 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tls-certs\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:41.867753 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:41.867730 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g47cz\" (UniqueName: \"kubernetes.io/projected/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-kube-api-access-g47cz\") pod \"e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:42.066152 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:42.066060 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:42.209330 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:42.209300 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6"] Jul 07 00:23:42.210953 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:23:42.210920 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb39e7d41_6423_42d3_82df_8cb6c8a30ac8.slice/crio-d4cc25ffd9290423257153716180123a3b44f956da12c6ce295773dc467716c3 WatchSource:0}: Error finding container d4cc25ffd9290423257153716180123a3b44f956da12c6ce295773dc467716c3: Status 404 returned error can't find the container with id d4cc25ffd9290423257153716180123a3b44f956da12c6ce295773dc467716c3 Jul 07 00:23:42.804795 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:42.804755 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" event={"ID":"b39e7d41-6423-42d3-82df-8cb6c8a30ac8","Type":"ContainerStarted","Data":"4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed"} Jul 07 00:23:42.804795 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:42.804800 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" event={"ID":"b39e7d41-6423-42d3-82df-8cb6c8a30ac8","Type":"ContainerStarted","Data":"d4cc25ffd9290423257153716180123a3b44f956da12c6ce295773dc467716c3"} Jul 07 00:23:42.823170 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:42.823118 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" podStartSLOduration=1.823099782 podStartE2EDuration="1.823099782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:23:42.822343098 +0000 UTC m=+1660.767928482" watchObservedRunningTime="2026-07-07 00:23:42.823099782 +0000 UTC m=+1660.768685153" Jul 07 00:23:46.059285 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:46.059240 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:23:51.158645 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:51.158592 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:51.258656 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:51.258623 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:52.066744 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:52.066714 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:52.067065 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:52.067048 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:52.079931 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:52.079902 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:23:52.108733 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:52.108699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:52.210090 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:52.210056 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:23:52.865630 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:23:52.865602 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:24:11.439785 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:11.439743 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:24:17.164555 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:17.164504 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:24:17.626144 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:17.626029 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:24:20.451560 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:20.451509 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:24:23.207079 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:23.207000 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:24:28.194273 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:28.194233 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6"] Jul 07 00:24:28.235636 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:28.235602 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:24:32.047865 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:32.047826 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:24:43.215498 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.215434 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" podUID="b39e7d41-6423-42d3-82df-8cb6c8a30ac8" containerName="main" containerID="cri-o://4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed" gracePeriod=45 Jul 07 00:24:43.483319 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.483291 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:24:43.618003 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.617963 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g47cz\" (UniqueName: \"kubernetes.io/projected/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-kube-api-access-g47cz\") pod \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " Jul 07 00:24:43.618003 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.618009 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-model-cache\") pod \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " Jul 07 00:24:43.618214 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.618062 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tls-certs\") pod \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " Jul 07 00:24:43.618214 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.618092 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-dshm\") pod \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " Jul 07 00:24:43.618214 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.618126 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-home\") pod \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " Jul 07 00:24:43.618214 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.618185 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tmp-dir\") pod \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\" (UID: \"b39e7d41-6423-42d3-82df-8cb6c8a30ac8\") " Jul 07 00:24:43.618394 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.618370 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-model-cache" (OuterVolumeSpecName: "model-cache") pod "b39e7d41-6423-42d3-82df-8cb6c8a30ac8" (UID: "b39e7d41-6423-42d3-82df-8cb6c8a30ac8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:43.618463 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.618406 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-home" (OuterVolumeSpecName: "home") pod "b39e7d41-6423-42d3-82df-8cb6c8a30ac8" (UID: "b39e7d41-6423-42d3-82df-8cb6c8a30ac8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:43.618598 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.618574 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b39e7d41-6423-42d3-82df-8cb6c8a30ac8" (UID: "b39e7d41-6423-42d3-82df-8cb6c8a30ac8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:43.620402 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.620365 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-dshm" (OuterVolumeSpecName: "dshm") pod "b39e7d41-6423-42d3-82df-8cb6c8a30ac8" (UID: "b39e7d41-6423-42d3-82df-8cb6c8a30ac8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:43.620516 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.620424 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b39e7d41-6423-42d3-82df-8cb6c8a30ac8" (UID: "b39e7d41-6423-42d3-82df-8cb6c8a30ac8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:24:43.620516 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.620443 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-kube-api-access-g47cz" (OuterVolumeSpecName: "kube-api-access-g47cz") pod "b39e7d41-6423-42d3-82df-8cb6c8a30ac8" (UID: "b39e7d41-6423-42d3-82df-8cb6c8a30ac8"). InnerVolumeSpecName "kube-api-access-g47cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:24:43.718863 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.718771 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tls-certs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:24:43.718863 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.718804 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-dshm\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:24:43.718863 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.718817 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-home\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:24:43.718863 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.718828 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-tmp-dir\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:24:43.718863 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.718840 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g47cz\" (UniqueName: \"kubernetes.io/projected/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-kube-api-access-g47cz\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:24:43.718863 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:43.718853 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b39e7d41-6423-42d3-82df-8cb6c8a30ac8-model-cache\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:24:44.105053 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.104962 2563 generic.go:356] "Generic (PLEG): container finished" podID="b39e7d41-6423-42d3-82df-8cb6c8a30ac8" containerID="4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed" exitCode=0 Jul 07 00:24:44.105053 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.105005 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" Jul 07 00:24:44.105053 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.105041 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" event={"ID":"b39e7d41-6423-42d3-82df-8cb6c8a30ac8","Type":"ContainerDied","Data":"4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed"} Jul 07 00:24:44.105321 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.105074 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6" event={"ID":"b39e7d41-6423-42d3-82df-8cb6c8a30ac8","Type":"ContainerDied","Data":"d4cc25ffd9290423257153716180123a3b44f956da12c6ce295773dc467716c3"} Jul 07 00:24:44.105321 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.105092 2563 scope.go:117] "RemoveContainer" containerID="4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed" Jul 07 00:24:44.115031 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.115010 2563 scope.go:117] "RemoveContainer" containerID="4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed" Jul 07 00:24:44.115355 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:24:44.115336 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed\": container with ID starting with 4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed not found: ID does not exist" containerID="4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed" Jul 07 00:24:44.115421 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.115370 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed"} err="failed to get container status \"4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed\": rpc error: code = NotFound desc = could not find container \"4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed\": container with ID starting with 4eb9693da67839e49ae4d6d3532a8c21a3ecd8cf3af0295436604979b9c297ed not found: ID does not exist" Jul 07 00:24:44.127881 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.127849 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6"] Jul 07 00:24:44.133840 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.133808 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7d49bf96/e2e-lifecycle-model-7d49bf96-kserve-6fcf8cb6ff-lz6p6"] Jul 07 00:24:44.576712 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:24:44.576679 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b39e7d41-6423-42d3-82df-8cb6c8a30ac8" path="/var/lib/kubelet/pods/b39e7d41-6423-42d3-82df-8cb6c8a30ac8/volumes" Jul 07 00:25:04.527339 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:04.527306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:05.590239 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:05.590210 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:07.348583 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:07.348542 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:25:11.184293 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:11.184256 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:11.280049 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:11.280016 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:11.468151 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:11.468061 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:11.691109 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:11.691073 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:12.776977 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:12.776943 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:12.888828 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:12.888796 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:12.986275 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:12.986241 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:13.778447 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:13.778418 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:13.877947 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:13.877904 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:13.977477 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:13.977442 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:15.396907 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:15.396864 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:16.360005 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:16.359958 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:25:21.072952 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:21.072916 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:21.139990 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:21.139956 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:21.746219 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:21.746185 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:27.845788 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:27.845745 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:27.950941 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:27.950907 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:28.042857 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:28.042818 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:28.453085 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:28.453054 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:25:34.053487 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:34.053452 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:38.595825 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:38.595789 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:38.931381 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:38.931346 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:25:59.910851 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:25:59.910813 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:05.979147 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:05.979112 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:06.068891 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:06.068860 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:06.298963 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:06.298875 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:06.626732 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:06.626639 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:06.797971 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:06.797932 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:09.819402 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:09.819363 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:09.903300 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:09.903259 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:09.983505 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:09.983472 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:10.327772 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:10.327733 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:10.524983 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:10.524948 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:10.636878 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:10.636842 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:14.321162 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:14.321123 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:14.651495 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:14.651461 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:14.963868 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:14.963776 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:15.063804 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:15.063772 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:15.949776 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:15.949740 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:26:20.851053 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:20.851008 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:20.944948 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:20.944916 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:21.168942 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:21.168908 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:21.360238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:21.360201 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:21.658045 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:21.658013 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:21.753102 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:21.753068 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:25.329917 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:25.329878 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:27.746957 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:27.746923 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:26:30.040341 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.040302 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh"] Jul 07 00:26:30.040772 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.040752 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b39e7d41-6423-42d3-82df-8cb6c8a30ac8" containerName="main" Jul 07 00:26:30.040822 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.040775 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b39e7d41-6423-42d3-82df-8cb6c8a30ac8" containerName="main" Jul 07 00:26:30.040887 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.040875 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b39e7d41-6423-42d3-82df-8cb6c8a30ac8" containerName="main" Jul 07 00:26:30.043866 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.043852 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.047319 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.047294 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cded59\"/\"default-dockercfg-8cq2m\"" Jul 07 00:26:30.047455 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.047301 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cded59\"/\"test-model-cded59-kserve-self-signed-certs\"" Jul 07 00:26:30.047455 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.047352 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cded59\"/\"kube-root-ca.crt\"" Jul 07 00:26:30.047455 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.047385 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cded59\"/\"openshift-service-ca.crt\"" Jul 07 00:26:30.053406 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.053385 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh"] Jul 07 00:26:30.157228 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.157192 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:30.164451 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.164415 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-dshm\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.164606 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.164456 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g99\" (UniqueName: \"kubernetes.io/projected/cf25293c-f20d-4a03-a087-b87997d7bf49-kube-api-access-s2g99\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.164606 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.164592 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-tmp-dir\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.164680 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.164627 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25293c-f20d-4a03-a087-b87997d7bf49-tls-certs\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.164718 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.164677 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-home\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.164718 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.164705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-model-cache\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.265526 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.265484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-tmp-dir\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.265526 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.265527 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25293c-f20d-4a03-a087-b87997d7bf49-tls-certs\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.265813 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.265586 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-home\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.265813 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.265608 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-model-cache\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.265813 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.265669 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-dshm\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.265813 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.265788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g99\" (UniqueName: \"kubernetes.io/projected/cf25293c-f20d-4a03-a087-b87997d7bf49-kube-api-access-s2g99\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.266052 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.266026 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-home\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.266179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.266022 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-tmp-dir\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.266179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.266127 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-model-cache\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.268046 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.268025 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-dshm\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.268245 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.268226 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25293c-f20d-4a03-a087-b87997d7bf49-tls-certs\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.275018 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.274984 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g99\" (UniqueName: \"kubernetes.io/projected/cf25293c-f20d-4a03-a087-b87997d7bf49-kube-api-access-s2g99\") pod \"test-model-cded59-kserve-67f97498c7-59ksh\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.354148 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.354047 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:30.494471 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.494433 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh"] Jul 07 00:26:30.496347 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:26:30.496310 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf25293c_f20d_4a03_a087_b87997d7bf49.slice/crio-151e34e54c7317ae1baa3b4860f875eb503ee21f2a6fdbdc0d02fcd053472794 WatchSource:0}: Error finding container 151e34e54c7317ae1baa3b4860f875eb503ee21f2a6fdbdc0d02fcd053472794: Status 404 returned error can't find the container with id 151e34e54c7317ae1baa3b4860f875eb503ee21f2a6fdbdc0d02fcd053472794 Jul 07 00:26:30.648271 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.648232 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" event={"ID":"cf25293c-f20d-4a03-a087-b87997d7bf49","Type":"ContainerStarted","Data":"1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db"} Jul 07 00:26:30.648443 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.648281 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" event={"ID":"cf25293c-f20d-4a03-a087-b87997d7bf49","Type":"ContainerStarted","Data":"151e34e54c7317ae1baa3b4860f875eb503ee21f2a6fdbdc0d02fcd053472794"} Jul 07 00:26:30.687506 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:30.687445 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" podStartSLOduration=0.687426075 podStartE2EDuration="687.426075ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:26:30.685968985 +0000 UTC m=+1828.631554357" watchObservedRunningTime="2026-07-07 00:26:30.687426075 +0000 UTC m=+1828.633011446" Jul 07 00:26:31.387321 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:31.387285 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:31.491669 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:31.491637 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:40.354420 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:40.354377 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:40.356909 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:40.354469 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:40.367051 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:40.367022 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:40.708328 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:40.708297 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:26:42.392197 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:42.387365 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:42.487265 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:42.487228 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:45.441905 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:45.441871 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:26:46.379817 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:46.379777 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:48.354142 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.354098 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt"] Jul 07 00:26:48.357651 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.357633 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.361463 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.361433 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0ec7c3\"/\"default-dockercfg-rqn49\"" Jul 07 00:26:48.361637 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.361439 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0ec7c3\"/\"openshift-service-ca.crt\"" Jul 07 00:26:48.361637 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.361443 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0ec7c3\"/\"kube-root-ca.crt\"" Jul 07 00:26:48.361637 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.361510 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0ec7c3\"/\"test-model-0ec7c3-kserve-self-signed-certs\"" Jul 07 00:26:48.366561 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.366502 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt"] Jul 07 00:26:48.536392 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.536348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-home\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.536636 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.536408 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-model-cache\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.536636 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.536438 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tmp-dir\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.536636 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.536464 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tls-certs\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.536636 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.536480 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-dshm\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.536636 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.536623 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btql4\" (UniqueName: \"kubernetes.io/projected/d3335705-cf6f-4218-9f9c-dd2eecb872bb-kube-api-access-btql4\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.637852 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.637748 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btql4\" (UniqueName: \"kubernetes.io/projected/d3335705-cf6f-4218-9f9c-dd2eecb872bb-kube-api-access-btql4\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.637852 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.637812 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-home\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.637852 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.637853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-model-cache\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.638154 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.637878 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tmp-dir\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.638154 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.637918 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tls-certs\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.638154 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.637941 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-dshm\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.638301 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.638278 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-home\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.638363 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.638342 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tmp-dir\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.638425 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.638369 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-model-cache\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.640402 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.640380 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-dshm\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.640804 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.640783 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tls-certs\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.645936 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.645913 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btql4\" (UniqueName: \"kubernetes.io/projected/d3335705-cf6f-4218-9f9c-dd2eecb872bb-kube-api-access-btql4\") pod \"test-model-0ec7c3-kserve-547445497f-nxxvt\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.668642 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.668599 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:48.808559 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:48.808510 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt"] Jul 07 00:26:48.811481 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:26:48.811453 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3335705_cf6f_4218_9f9c_dd2eecb872bb.slice/crio-e3fdda6c840fbba9592eb63b632813c132d846b206c4b181c11786186025ff5a WatchSource:0}: Error finding container e3fdda6c840fbba9592eb63b632813c132d846b206c4b181c11786186025ff5a: Status 404 returned error can't find the container with id e3fdda6c840fbba9592eb63b632813c132d846b206c4b181c11786186025ff5a Jul 07 00:26:49.067367 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:49.067316 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:26:49.746922 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:49.746885 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" event={"ID":"d3335705-cf6f-4218-9f9c-dd2eecb872bb","Type":"ContainerStarted","Data":"56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062"} Jul 07 00:26:49.746922 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:49.746924 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" event={"ID":"d3335705-cf6f-4218-9f9c-dd2eecb872bb","Type":"ContainerStarted","Data":"e3fdda6c840fbba9592eb63b632813c132d846b206c4b181c11786186025ff5a"} Jul 07 00:26:49.766647 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:49.766593 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" podStartSLOduration=1.766574398 podStartE2EDuration="1.766574398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:26:49.764665341 +0000 UTC m=+1847.710250715" watchObservedRunningTime="2026-07-07 00:26:49.766574398 +0000 UTC m=+1847.712159770" Jul 07 00:26:56.145961 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:56.145928 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:26:58.669561 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:58.669512 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:58.669561 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:58.669571 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:58.682122 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:58.682100 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:26:58.807312 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:26:58.807280 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:27:13.508372 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:13.508335 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh"] Jul 07 00:27:13.531613 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:13.531580 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:14.487486 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:14.487445 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt"] Jul 07 00:27:15.378010 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:15.377976 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:15.446718 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:15.446678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:18.243568 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:18.243520 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:27:20.133212 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:20.133173 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:28.529510 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.529384 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" podUID="cf25293c-f20d-4a03-a087-b87997d7bf49" containerName="main" containerID="cri-o://1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db" gracePeriod=45 Jul 07 00:27:28.786828 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.786760 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:27:28.823831 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.823794 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-tmp-dir\") pod \"cf25293c-f20d-4a03-a087-b87997d7bf49\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " Jul 07 00:27:28.824026 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.823840 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-dshm\") pod \"cf25293c-f20d-4a03-a087-b87997d7bf49\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " Jul 07 00:27:28.824026 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.823873 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2g99\" (UniqueName: \"kubernetes.io/projected/cf25293c-f20d-4a03-a087-b87997d7bf49-kube-api-access-s2g99\") pod \"cf25293c-f20d-4a03-a087-b87997d7bf49\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " Jul 07 00:27:28.824026 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.823898 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-home\") pod \"cf25293c-f20d-4a03-a087-b87997d7bf49\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " Jul 07 00:27:28.824026 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.823924 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-model-cache\") pod \"cf25293c-f20d-4a03-a087-b87997d7bf49\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " Jul 07 00:27:28.824026 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.823975 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25293c-f20d-4a03-a087-b87997d7bf49-tls-certs\") pod \"cf25293c-f20d-4a03-a087-b87997d7bf49\" (UID: \"cf25293c-f20d-4a03-a087-b87997d7bf49\") " Jul 07 00:27:28.824292 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.824137 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-home" (OuterVolumeSpecName: "home") pod "cf25293c-f20d-4a03-a087-b87997d7bf49" (UID: "cf25293c-f20d-4a03-a087-b87997d7bf49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:28.824292 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.824168 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf25293c-f20d-4a03-a087-b87997d7bf49" (UID: "cf25293c-f20d-4a03-a087-b87997d7bf49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:28.824402 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.824295 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-home\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:28.824402 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.824314 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-tmp-dir\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:28.824962 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.824921 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf25293c-f20d-4a03-a087-b87997d7bf49" (UID: "cf25293c-f20d-4a03-a087-b87997d7bf49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:28.826394 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.826362 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-dshm" (OuterVolumeSpecName: "dshm") pod "cf25293c-f20d-4a03-a087-b87997d7bf49" (UID: "cf25293c-f20d-4a03-a087-b87997d7bf49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:28.826516 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.826400 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf25293c-f20d-4a03-a087-b87997d7bf49-kube-api-access-s2g99" (OuterVolumeSpecName: "kube-api-access-s2g99") pod "cf25293c-f20d-4a03-a087-b87997d7bf49" (UID: "cf25293c-f20d-4a03-a087-b87997d7bf49"). InnerVolumeSpecName "kube-api-access-s2g99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:27:28.826516 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.826420 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf25293c-f20d-4a03-a087-b87997d7bf49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf25293c-f20d-4a03-a087-b87997d7bf49" (UID: "cf25293c-f20d-4a03-a087-b87997d7bf49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:27:28.925156 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.925115 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-dshm\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:28.925156 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.925150 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2g99\" (UniqueName: \"kubernetes.io/projected/cf25293c-f20d-4a03-a087-b87997d7bf49-kube-api-access-s2g99\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:28.925375 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.925167 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25293c-f20d-4a03-a087-b87997d7bf49-model-cache\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:28.925375 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.925183 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25293c-f20d-4a03-a087-b87997d7bf49-tls-certs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:28.957657 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.957619 2563 generic.go:356] "Generic (PLEG): container finished" podID="cf25293c-f20d-4a03-a087-b87997d7bf49" containerID="1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db" exitCode=0 Jul 07 00:27:28.957857 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.957666 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" Jul 07 00:27:28.957857 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.957706 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" event={"ID":"cf25293c-f20d-4a03-a087-b87997d7bf49","Type":"ContainerDied","Data":"1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db"} Jul 07 00:27:28.957857 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.957760 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh" event={"ID":"cf25293c-f20d-4a03-a087-b87997d7bf49","Type":"ContainerDied","Data":"151e34e54c7317ae1baa3b4860f875eb503ee21f2a6fdbdc0d02fcd053472794"} Jul 07 00:27:28.957857 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.957782 2563 scope.go:117] "RemoveContainer" containerID="1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db" Jul 07 00:27:28.975116 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.975089 2563 scope.go:117] "RemoveContainer" containerID="1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db" Jul 07 00:27:28.975438 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:27:28.975407 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db\": container with ID starting with 1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db not found: ID does not exist" containerID="1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db" Jul 07 00:27:28.975559 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.975442 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db"} err="failed to get container status \"1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db\": rpc error: code = NotFound desc = could not find container \"1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db\": container with ID starting with 1c74a0974e9aabae3ef78847a74e96b02623a25777dfec2e037dc260606dd3db not found: ID does not exist" Jul 07 00:27:28.985391 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.985362 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh"] Jul 07 00:27:28.988825 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:28.988800 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cded59/test-model-cded59-kserve-67f97498c7-59ksh"] Jul 07 00:27:29.508689 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.508644 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" podUID="d3335705-cf6f-4218-9f9c-dd2eecb872bb" containerName="main" containerID="cri-o://56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062" gracePeriod=45 Jul 07 00:27:29.759518 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.759452 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:27:29.830869 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.830833 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btql4\" (UniqueName: \"kubernetes.io/projected/d3335705-cf6f-4218-9f9c-dd2eecb872bb-kube-api-access-btql4\") pod \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " Jul 07 00:27:29.830869 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.830884 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-dshm\") pod \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " Jul 07 00:27:29.831206 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.830940 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-model-cache\") pod \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " Jul 07 00:27:29.831206 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.830994 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tls-certs\") pod \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " Jul 07 00:27:29.831206 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.831053 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tmp-dir\") pod \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " Jul 07 00:27:29.831206 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.831097 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-home\") pod \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\" (UID: \"d3335705-cf6f-4218-9f9c-dd2eecb872bb\") " Jul 07 00:27:29.831404 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.831295 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3335705-cf6f-4218-9f9c-dd2eecb872bb" (UID: "d3335705-cf6f-4218-9f9c-dd2eecb872bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:29.831404 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.831355 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3335705-cf6f-4218-9f9c-dd2eecb872bb" (UID: "d3335705-cf6f-4218-9f9c-dd2eecb872bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:29.831404 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.831363 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-home" (OuterVolumeSpecName: "home") pod "d3335705-cf6f-4218-9f9c-dd2eecb872bb" (UID: "d3335705-cf6f-4218-9f9c-dd2eecb872bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:29.831561 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.831420 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-model-cache\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:29.833459 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.833426 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3335705-cf6f-4218-9f9c-dd2eecb872bb" (UID: "d3335705-cf6f-4218-9f9c-dd2eecb872bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:27:29.833670 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.833654 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-dshm" (OuterVolumeSpecName: "dshm") pod "d3335705-cf6f-4218-9f9c-dd2eecb872bb" (UID: "d3335705-cf6f-4218-9f9c-dd2eecb872bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:29.833754 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.833726 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3335705-cf6f-4218-9f9c-dd2eecb872bb-kube-api-access-btql4" (OuterVolumeSpecName: "kube-api-access-btql4") pod "d3335705-cf6f-4218-9f9c-dd2eecb872bb" (UID: "d3335705-cf6f-4218-9f9c-dd2eecb872bb"). InnerVolumeSpecName "kube-api-access-btql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:27:29.932605 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.932564 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tmp-dir\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:29.932605 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.932599 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-home\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:29.932816 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.932611 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btql4\" (UniqueName: \"kubernetes.io/projected/d3335705-cf6f-4218-9f9c-dd2eecb872bb-kube-api-access-btql4\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:29.932816 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.932627 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3335705-cf6f-4218-9f9c-dd2eecb872bb-dshm\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:29.932816 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.932639 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3335705-cf6f-4218-9f9c-dd2eecb872bb-tls-certs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:27:29.964830 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.964791 2563 generic.go:356] "Generic (PLEG): container finished" podID="d3335705-cf6f-4218-9f9c-dd2eecb872bb" containerID="56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062" exitCode=0 Jul 07 00:27:29.964976 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.964836 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" Jul 07 00:27:29.964976 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.964880 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" event={"ID":"d3335705-cf6f-4218-9f9c-dd2eecb872bb","Type":"ContainerDied","Data":"56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062"} Jul 07 00:27:29.964976 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.964923 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt" event={"ID":"d3335705-cf6f-4218-9f9c-dd2eecb872bb","Type":"ContainerDied","Data":"e3fdda6c840fbba9592eb63b632813c132d846b206c4b181c11786186025ff5a"} Jul 07 00:27:29.964976 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.964942 2563 scope.go:117] "RemoveContainer" containerID="56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062" Jul 07 00:27:29.974624 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.974600 2563 scope.go:117] "RemoveContainer" containerID="56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062" Jul 07 00:27:29.974908 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:27:29.974889 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062\": container with ID starting with 56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062 not found: ID does not exist" containerID="56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062" Jul 07 00:27:29.974955 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.974919 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062"} err="failed to get container status \"56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062\": rpc error: code = NotFound desc = could not find container \"56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062\": container with ID starting with 56c235e4db6a9361f0f8834aff3771edc11f384c41b9818cec10c27ada396062 not found: ID does not exist" Jul 07 00:27:29.985380 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.985348 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt"] Jul 07 00:27:29.989716 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:29.989687 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0ec7c3/test-model-0ec7c3-kserve-547445497f-nxxvt"] Jul 07 00:27:30.576458 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:30.576422 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf25293c-f20d-4a03-a087-b87997d7bf49" path="/var/lib/kubelet/pods/cf25293c-f20d-4a03-a087-b87997d7bf49/volumes" Jul 07 00:27:30.576862 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:30.576846 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3335705-cf6f-4218-9f9c-dd2eecb872bb" path="/var/lib/kubelet/pods/d3335705-cf6f-4218-9f9c-dd2eecb872bb/volumes" Jul 07 00:27:51.685273 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:51.685233 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:57.153447 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:57.153407 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:57.436804 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:57.436703 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:57.653077 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:57.653043 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:57.862181 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:57.862084 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:58.221692 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:58.221657 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:58.376119 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:58.376068 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:58.476478 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:58.476385 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:27:58.619813 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:27:58.619781 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:03.947555 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:03.947498 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:28:04.049853 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:04.049815 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:04.150880 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:04.150849 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:04.321231 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:04.321148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:04.561753 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:04.561721 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:04.872135 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:04.872096 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:05.198842 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:05.198800 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:05.393663 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:05.393629 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:06.734225 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:06.734189 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:06.789294 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:06.789266 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:08.344862 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:08.344821 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:08.440324 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:08.440286 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:08.543849 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:08.543810 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:08.679970 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:08.679933 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:09.154235 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:09.154203 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:09.270246 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:09.270217 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:12.833479 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:12.833447 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:13.273470 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:13.273438 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:13.594230 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:13.594106 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:13.704491 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:13.704460 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:15.144294 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:15.144254 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:28:16.351050 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:16.351014 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:16.455204 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:16.455158 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:16.551785 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:16.551750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:16.672578 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:16.672545 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:17.234645 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:17.234606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:17.477993 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:17.477965 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:20.862039 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:20.862009 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:26.473480 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:26.473445 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:28:28.974850 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:28.974818 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:29.484650 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:29.484616 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:29.849986 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:29.849893 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:29.947551 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:29.947495 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:30.151066 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:30.151033 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:35.512836 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:35.512790 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:35.586473 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:35.586426 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:35.779561 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:35.779435 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:36.015678 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:36.015648 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:36.199814 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:36.199773 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:36.619158 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:36.619053 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:36.904013 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:36.903947 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:37.033399 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:37.033369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:37.193152 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:37.193068 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:41.250196 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:41.250161 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:28:42.420958 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:42.420919 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:42.490380 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:42.490346 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:42.701010 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:42.700918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:42.912457 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:42.912423 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:43.251057 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:43.251019 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:43.573387 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:43.573277 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:43.810270 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:43.810234 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:46.749675 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:46.749643 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:46.914754 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:46.914723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:47.052614 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:47.052509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:47.448959 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:47.448925 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:47.548050 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:47.548013 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:47.648898 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:47.648862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:51.084913 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:51.084880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:51.490798 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:51.490768 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:51.711787 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:51.711745 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:51.849579 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:51.849458 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:28:51.989725 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:51.989680 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:52.090152 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:52.090122 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:52.172511 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:52.172476 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:54.541438 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:54.541374 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:54.619518 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:54.619474 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:54.692734 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:54.692699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:55.026013 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:55.025983 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:55.330052 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:55.329961 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:55.429943 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:55.429905 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:28:59.037881 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:28:59.037843 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:29:00.547513 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:00.547477 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:29:08.292176 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:08.292133 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:29:08.393345 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:08.393313 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:29:16.052978 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:16.052938 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:29:16.153897 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:16.153862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:29:17.168850 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.168810 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq"] Jul 07 00:29:17.169239 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.169165 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3335705-cf6f-4218-9f9c-dd2eecb872bb" containerName="main" Jul 07 00:29:17.169239 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.169177 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3335705-cf6f-4218-9f9c-dd2eecb872bb" containerName="main" Jul 07 00:29:17.169239 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.169185 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf25293c-f20d-4a03-a087-b87997d7bf49" containerName="main" Jul 07 00:29:17.169239 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.169190 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf25293c-f20d-4a03-a087-b87997d7bf49" containerName="main" Jul 07 00:29:17.169365 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.169244 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3335705-cf6f-4218-9f9c-dd2eecb872bb" containerName="main" Jul 07 00:29:17.169365 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.169253 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf25293c-f20d-4a03-a087-b87997d7bf49" containerName="main" Jul 07 00:29:17.171914 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.171897 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.175936 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.175909 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"default-dockercfg-ngv9s\"" Jul 07 00:29:17.176103 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.175935 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"kube-root-ca.crt\"" Jul 07 00:29:17.176103 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.175926 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"openshift-service-ca.crt\"" Jul 07 00:29:17.176103 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.175943 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"auth-test-model-8714ac-kserve-self-signed-certs\"" Jul 07 00:29:17.176277 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.176258 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-dshm\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.176314 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.176296 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tmp-dir\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.176363 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.176351 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-home\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.176413 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.176396 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp995\" (UniqueName: \"kubernetes.io/projected/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-kube-api-access-lp995\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.176582 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.176566 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-model-cache\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.176630 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.176596 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tls-certs\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.181893 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.181867 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq"] Jul 07 00:29:17.253056 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.253024 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:29:17.277619 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.277585 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-model-cache\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.277619 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.277623 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tls-certs\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.277845 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.277649 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-dshm\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.277845 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.277769 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tmp-dir\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.277845 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.277809 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-home\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.278005 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.277846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp995\" (UniqueName: \"kubernetes.io/projected/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-kube-api-access-lp995\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.278128 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.278098 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-model-cache\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.278252 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.278148 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tmp-dir\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.278252 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.278170 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-home\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.280088 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.280065 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-dshm\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.280181 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.280161 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tls-certs\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.287113 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.287088 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp995\" (UniqueName: \"kubernetes.io/projected/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-kube-api-access-lp995\") pod \"auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.481711 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.481619 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:17.622423 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:17.622393 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq"] Jul 07 00:29:17.623084 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:29:17.623061 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a2bbe45_73e0_46b1_ad2f_665f0912f1dc.slice/crio-a8ce188ed64a922ed08e09590839d23a8d48f91af571c5583daf221c9995a337 WatchSource:0}: Error finding container a8ce188ed64a922ed08e09590839d23a8d48f91af571c5583daf221c9995a337: Status 404 returned error can't find the container with id a8ce188ed64a922ed08e09590839d23a8d48f91af571c5583daf221c9995a337 Jul 07 00:29:18.520551 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:18.520495 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" event={"ID":"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc","Type":"ContainerStarted","Data":"b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696"} Jul 07 00:29:18.520551 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:18.520550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" event={"ID":"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc","Type":"ContainerStarted","Data":"a8ce188ed64a922ed08e09590839d23a8d48f91af571c5583daf221c9995a337"} Jul 07 00:29:18.541624 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:18.541561 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" podStartSLOduration=1.541542215 podStartE2EDuration="1.541542215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:29:18.539411716 +0000 UTC m=+1996.484997091" watchObservedRunningTime="2026-07-07 00:29:18.541542215 +0000 UTC m=+1996.487127577" Jul 07 00:29:20.237402 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:20.237365 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:29:27.482148 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:27.482099 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:27.482586 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:27.482172 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:27.494753 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:27.494724 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:27.579843 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:27.579812 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:29:46.275040 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.275006 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc"] Jul 07 00:29:46.277819 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.277801 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.281850 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.281824 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"openshift-service-ca.crt\"" Jul 07 00:29:46.282086 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.282068 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"default-dockercfg-dv9q9\"" Jul 07 00:29:46.282192 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.282156 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"kube-root-ca.crt\"" Jul 07 00:29:46.282288 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.282274 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"auth-test-model-4e3cf5-kserve-self-signed-certs\"" Jul 07 00:29:46.287796 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.287772 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc"] Jul 07 00:29:46.294037 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.294008 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:29:46.338316 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.338275 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7z4r\" (UniqueName: \"kubernetes.io/projected/919e991b-1f91-4d9b-a9ac-0443e5160070-kube-api-access-m7z4r\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.338496 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.338336 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-dshm\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.338496 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.338383 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-home\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.338496 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.338406 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919e991b-1f91-4d9b-a9ac-0443e5160070-tls-certs\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.338496 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.338436 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-model-cache\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.338496 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.338480 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-tmp-dir\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.439802 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.439769 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-dshm\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.440014 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.439825 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-home\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.440014 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.439845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919e991b-1f91-4d9b-a9ac-0443e5160070-tls-certs\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.440014 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.439970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-model-cache\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.440197 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.440016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-tmp-dir\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.440197 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.440097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7z4r\" (UniqueName: \"kubernetes.io/projected/919e991b-1f91-4d9b-a9ac-0443e5160070-kube-api-access-m7z4r\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.440301 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.440255 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-home\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.440390 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.440368 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-model-cache\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.440449 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.440380 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-tmp-dir\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.442378 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.442356 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-dshm\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.442651 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.442629 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919e991b-1f91-4d9b-a9ac-0443e5160070-tls-certs\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.449378 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.449355 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7z4r\" (UniqueName: \"kubernetes.io/projected/919e991b-1f91-4d9b-a9ac-0443e5160070-kube-api-access-m7z4r\") pod \"auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.588507 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.588419 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:46.729218 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:46.729189 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc"] Jul 07 00:29:46.731056 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:29:46.731014 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod919e991b_1f91_4d9b_a9ac_0443e5160070.slice/crio-7c3ac9a3fb5fed18fd87795deed5d028646d1c5e804cc1551b49fc936db49064 WatchSource:0}: Error finding container 7c3ac9a3fb5fed18fd87795deed5d028646d1c5e804cc1551b49fc936db49064: Status 404 returned error can't find the container with id 7c3ac9a3fb5fed18fd87795deed5d028646d1c5e804cc1551b49fc936db49064 Jul 07 00:29:47.671309 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:47.671269 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" event={"ID":"919e991b-1f91-4d9b-a9ac-0443e5160070","Type":"ContainerStarted","Data":"2feb03d41a5f9da35dc59639e6b6f5754ea0fcb854ebe4ca4c829936f92bda9f"} Jul 07 00:29:47.671309 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:47.671316 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" event={"ID":"919e991b-1f91-4d9b-a9ac-0443e5160070","Type":"ContainerStarted","Data":"7c3ac9a3fb5fed18fd87795deed5d028646d1c5e804cc1551b49fc936db49064"} Jul 07 00:29:47.700487 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:47.700419 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" podStartSLOduration=1.700402801 podStartE2EDuration="1.700402801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:29:47.700057926 +0000 UTC m=+2025.645643326" watchObservedRunningTime="2026-07-07 00:29:47.700402801 +0000 UTC m=+2025.645988172" Jul 07 00:29:49.945984 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:49.945945 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:29:56.589084 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:56.589045 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:56.589476 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:56.589095 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:56.602347 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:56.602315 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:29:56.736785 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:29:56.736754 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:30:07.846262 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:30:07.846217 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:30:21.249294 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:30:21.249259 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:30:36.949412 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:30:36.949329 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:30:52.440863 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:30:52.440830 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:31:04.750311 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:31:04.750269 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:31:23.053086 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:31:23.053046 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:31:36.042277 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:31:36.042245 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:31:49.484663 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:31:49.484630 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq"] Jul 07 00:31:49.586746 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:31:49.586706 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:31:50.246039 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:31:50.245999 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc"] Jul 07 00:31:50.269188 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:31:50.269151 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:31:52.443366 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:31:52.443332 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:32:04.505252 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.505183 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" podUID="1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" containerName="main" containerID="cri-o://b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696" gracePeriod=45 Jul 07 00:32:04.754926 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.754900 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:32:04.826063 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.825964 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-dshm\") pod \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " Jul 07 00:32:04.826063 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.826012 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-home\") pod \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " Jul 07 00:32:04.826063 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.826041 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp995\" (UniqueName: \"kubernetes.io/projected/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-kube-api-access-lp995\") pod \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " Jul 07 00:32:04.826359 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.826070 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tls-certs\") pod \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " Jul 07 00:32:04.826359 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.826098 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tmp-dir\") pod \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " Jul 07 00:32:04.826359 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.826181 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-model-cache\") pod \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\" (UID: \"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc\") " Jul 07 00:32:04.826581 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.826361 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-home" (OuterVolumeSpecName: "home") pod "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" (UID: "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:32:04.826581 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.826457 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" (UID: "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:32:04.826581 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.826487 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-home\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:04.826581 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.826477 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" (UID: "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:32:04.828442 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.828419 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" (UID: "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:32:04.828598 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.828460 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-dshm" (OuterVolumeSpecName: "dshm") pod "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" (UID: "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:32:04.828598 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.828510 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-kube-api-access-lp995" (OuterVolumeSpecName: "kube-api-access-lp995") pod "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" (UID: "1a2bbe45-73e0-46b1-ad2f-665f0912f1dc"). InnerVolumeSpecName "kube-api-access-lp995". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:32:04.927706 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.927660 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-model-cache\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:04.927706 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.927697 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-dshm\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:04.927706 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.927707 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp995\" (UniqueName: \"kubernetes.io/projected/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-kube-api-access-lp995\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:04.927706 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.927716 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tls-certs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:04.927988 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:04.927726 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc-tmp-dir\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:05.267198 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.267159 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" podUID="919e991b-1f91-4d9b-a9ac-0443e5160070" containerName="main" containerID="cri-o://2feb03d41a5f9da35dc59639e6b6f5754ea0fcb854ebe4ca4c829936f92bda9f" gracePeriod=45 Jul 07 00:32:05.405072 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.405037 2563 generic.go:356] "Generic (PLEG): container finished" podID="1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" containerID="b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696" exitCode=0 Jul 07 00:32:05.405242 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.405086 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" Jul 07 00:32:05.405242 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.405124 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" event={"ID":"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc","Type":"ContainerDied","Data":"b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696"} Jul 07 00:32:05.405242 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.405170 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq" event={"ID":"1a2bbe45-73e0-46b1-ad2f-665f0912f1dc","Type":"ContainerDied","Data":"a8ce188ed64a922ed08e09590839d23a8d48f91af571c5583daf221c9995a337"} Jul 07 00:32:05.405242 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.405194 2563 scope.go:117] "RemoveContainer" containerID="b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696" Jul 07 00:32:05.407384 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.407347 2563 generic.go:356] "Generic (PLEG): container finished" podID="919e991b-1f91-4d9b-a9ac-0443e5160070" containerID="2feb03d41a5f9da35dc59639e6b6f5754ea0fcb854ebe4ca4c829936f92bda9f" exitCode=0 Jul 07 00:32:05.407518 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.407419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" event={"ID":"919e991b-1f91-4d9b-a9ac-0443e5160070","Type":"ContainerDied","Data":"2feb03d41a5f9da35dc59639e6b6f5754ea0fcb854ebe4ca4c829936f92bda9f"} Jul 07 00:32:05.415764 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.415738 2563 scope.go:117] "RemoveContainer" containerID="b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696" Jul 07 00:32:05.416089 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:32:05.416069 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696\": container with ID starting with b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696 not found: ID does not exist" containerID="b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696" Jul 07 00:32:05.416147 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.416099 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696"} err="failed to get container status \"b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696\": rpc error: code = NotFound desc = could not find container \"b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696\": container with ID starting with b90e387fed679540a3623cab9a6285a958513ea7c1113dae0a32dd96cf2d3696 not found: ID does not exist" Jul 07 00:32:05.427902 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.427869 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq"] Jul 07 00:32:05.431907 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.431873 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/auth-test-model-8714ac-kserve-6f54dc9c47-kdwjq"] Jul 07 00:32:05.507736 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.507710 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:32:05.570336 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.570298 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp"] Jul 07 00:32:05.570851 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.570832 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" containerName="main" Jul 07 00:32:05.570906 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.570856 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" containerName="main" Jul 07 00:32:05.570906 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.570872 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919e991b-1f91-4d9b-a9ac-0443e5160070" containerName="main" Jul 07 00:32:05.570906 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.570879 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="919e991b-1f91-4d9b-a9ac-0443e5160070" containerName="main" Jul 07 00:32:05.571008 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.570945 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="919e991b-1f91-4d9b-a9ac-0443e5160070" containerName="main" Jul 07 00:32:05.571008 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.570955 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" containerName="main" Jul 07 00:32:05.573705 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.573690 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.576942 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.576910 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"sub-test-model-8714ac-kserve-self-signed-certs\"" Jul 07 00:32:05.576942 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.576930 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"openshift-service-ca.crt\"" Jul 07 00:32:05.577113 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.577043 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"default-dockercfg-ngv9s\"" Jul 07 00:32:05.577960 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.577942 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"kube-root-ca.crt\"" Jul 07 00:32:05.585304 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.585279 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp"] Jul 07 00:32:05.635276 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.635242 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7z4r\" (UniqueName: \"kubernetes.io/projected/919e991b-1f91-4d9b-a9ac-0443e5160070-kube-api-access-m7z4r\") pod \"919e991b-1f91-4d9b-a9ac-0443e5160070\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " Jul 07 00:32:05.635452 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.635304 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919e991b-1f91-4d9b-a9ac-0443e5160070-tls-certs\") pod \"919e991b-1f91-4d9b-a9ac-0443e5160070\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " Jul 07 00:32:05.635452 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.635330 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-dshm\") pod \"919e991b-1f91-4d9b-a9ac-0443e5160070\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " Jul 07 00:32:05.635452 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.635384 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-model-cache\") pod \"919e991b-1f91-4d9b-a9ac-0443e5160070\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " Jul 07 00:32:05.635452 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.635417 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-tmp-dir\") pod \"919e991b-1f91-4d9b-a9ac-0443e5160070\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " Jul 07 00:32:05.635452 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.635440 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-home\") pod \"919e991b-1f91-4d9b-a9ac-0443e5160070\" (UID: \"919e991b-1f91-4d9b-a9ac-0443e5160070\") " Jul 07 00:32:05.635765 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.635724 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-model-cache" (OuterVolumeSpecName: "model-cache") pod "919e991b-1f91-4d9b-a9ac-0443e5160070" (UID: "919e991b-1f91-4d9b-a9ac-0443e5160070"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:32:05.635765 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.635737 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "919e991b-1f91-4d9b-a9ac-0443e5160070" (UID: "919e991b-1f91-4d9b-a9ac-0443e5160070"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:32:05.635959 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.635936 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-home" (OuterVolumeSpecName: "home") pod "919e991b-1f91-4d9b-a9ac-0443e5160070" (UID: "919e991b-1f91-4d9b-a9ac-0443e5160070"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:32:05.637778 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.637748 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-dshm" (OuterVolumeSpecName: "dshm") pod "919e991b-1f91-4d9b-a9ac-0443e5160070" (UID: "919e991b-1f91-4d9b-a9ac-0443e5160070"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:32:05.637895 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.637784 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919e991b-1f91-4d9b-a9ac-0443e5160070-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "919e991b-1f91-4d9b-a9ac-0443e5160070" (UID: "919e991b-1f91-4d9b-a9ac-0443e5160070"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:32:05.637895 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.637874 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919e991b-1f91-4d9b-a9ac-0443e5160070-kube-api-access-m7z4r" (OuterVolumeSpecName: "kube-api-access-m7z4r") pod "919e991b-1f91-4d9b-a9ac-0443e5160070" (UID: "919e991b-1f91-4d9b-a9ac-0443e5160070"). InnerVolumeSpecName "kube-api-access-m7z4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:32:05.648916 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.648888 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:32:05.672026 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.671998 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:32:05.736435 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736393 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-home\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.736435 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736435 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-dshm\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.736713 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736467 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f040b410-0c36-4d2f-af28-25e19e62660f-tls-certs\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.736713 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736635 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-tmp-dir\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.736713 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bd5\" (UniqueName: \"kubernetes.io/projected/f040b410-0c36-4d2f-af28-25e19e62660f-kube-api-access-r2bd5\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.736910 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736738 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-model-cache\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.736910 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736821 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-dshm\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:05.736910 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736845 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-model-cache\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:05.736910 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736882 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-tmp-dir\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:05.736910 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736901 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919e991b-1f91-4d9b-a9ac-0443e5160070-home\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:05.737078 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736917 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7z4r\" (UniqueName: \"kubernetes.io/projected/919e991b-1f91-4d9b-a9ac-0443e5160070-kube-api-access-m7z4r\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:05.737078 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.736932 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919e991b-1f91-4d9b-a9ac-0443e5160070-tls-certs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:32:05.838372 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.838268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-home\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.838372 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.838307 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-dshm\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.838372 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.838332 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f040b410-0c36-4d2f-af28-25e19e62660f-tls-certs\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.838721 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.838386 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-tmp-dir\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.838721 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.838444 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bd5\" (UniqueName: \"kubernetes.io/projected/f040b410-0c36-4d2f-af28-25e19e62660f-kube-api-access-r2bd5\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.838721 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.838592 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-model-cache\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.838879 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.838802 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-home\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.838879 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.838868 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-tmp-dir\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.838985 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.838912 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-model-cache\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.840890 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.840873 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-dshm\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.841171 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.841150 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f040b410-0c36-4d2f-af28-25e19e62660f-tls-certs\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.847508 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.847483 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bd5\" (UniqueName: \"kubernetes.io/projected/f040b410-0c36-4d2f-af28-25e19e62660f-kube-api-access-r2bd5\") pod \"sub-test-model-8714ac-kserve-749fb4bf99-hhsqp\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:05.884037 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:05.883999 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:06.027316 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.027282 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp"] Jul 07 00:32:06.030662 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:32:06.030517 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf040b410_0c36_4d2f_af28_25e19e62660f.slice/crio-a302844f3f435f69316cb04ea59d516a456d13399f3e50bbd7d9a6d037df8f55 WatchSource:0}: Error finding container a302844f3f435f69316cb04ea59d516a456d13399f3e50bbd7d9a6d037df8f55: Status 404 returned error can't find the container with id a302844f3f435f69316cb04ea59d516a456d13399f3e50bbd7d9a6d037df8f55 Jul 07 00:32:06.315166 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.315123 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:32:06.413762 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.413722 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" event={"ID":"f040b410-0c36-4d2f-af28-25e19e62660f","Type":"ContainerStarted","Data":"589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3"} Jul 07 00:32:06.413972 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.413772 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" event={"ID":"f040b410-0c36-4d2f-af28-25e19e62660f","Type":"ContainerStarted","Data":"a302844f3f435f69316cb04ea59d516a456d13399f3e50bbd7d9a6d037df8f55"} Jul 07 00:32:06.415806 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.415784 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" event={"ID":"919e991b-1f91-4d9b-a9ac-0443e5160070","Type":"ContainerDied","Data":"7c3ac9a3fb5fed18fd87795deed5d028646d1c5e804cc1551b49fc936db49064"} Jul 07 00:32:06.415936 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.415821 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc" Jul 07 00:32:06.415936 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.415825 2563 scope.go:117] "RemoveContainer" containerID="2feb03d41a5f9da35dc59639e6b6f5754ea0fcb854ebe4ca4c829936f92bda9f" Jul 07 00:32:06.433967 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.433913 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" podStartSLOduration=1.433897637 podStartE2EDuration="1.433897637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:32:06.431141622 +0000 UTC m=+2164.376726994" watchObservedRunningTime="2026-07-07 00:32:06.433897637 +0000 UTC m=+2164.379483008" Jul 07 00:32:06.444337 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.444303 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc"] Jul 07 00:32:06.452687 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.452656 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/auth-test-model-4e3cf5-kserve-7cd7f64b47-6zcxc"] Jul 07 00:32:06.576901 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.576820 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a2bbe45-73e0-46b1-ad2f-665f0912f1dc" path="/var/lib/kubelet/pods/1a2bbe45-73e0-46b1-ad2f-665f0912f1dc/volumes" Jul 07 00:32:06.577267 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:06.577212 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919e991b-1f91-4d9b-a9ac-0443e5160070" path="/var/lib/kubelet/pods/919e991b-1f91-4d9b-a9ac-0443e5160070/volumes" Jul 07 00:32:15.884279 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:15.884240 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:15.884860 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:15.884297 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:15.897182 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:15.897153 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:16.348961 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:16.348921 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:32:16.477134 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:16.477099 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:32:33.371697 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.371659 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8"] Jul 07 00:32:33.376458 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.376435 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.380362 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.380337 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"kube-root-ca.crt\"" Jul 07 00:32:33.380362 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.380337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"default-dockercfg-dv9q9\"" Jul 07 00:32:33.380599 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.380380 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"openshift-service-ca.crt\"" Jul 07 00:32:33.380599 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.380338 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"sub-test-model-4e3cf5-kserve-self-signed-certs\"" Jul 07 00:32:33.387989 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.387938 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8"] Jul 07 00:32:33.394765 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.394733 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:32:33.481988 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.481944 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-kube-api-access-z8bjc\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.482179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.482001 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tmp-dir\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.482179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.482040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tls-certs\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.482179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.482109 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-dshm\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.482179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.482143 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-home\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.482179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.482158 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-model-cache\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.583083 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.583020 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-kube-api-access-z8bjc\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.583083 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.583091 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tmp-dir\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.583361 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.583119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tls-certs\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.583361 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.583163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-dshm\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.583361 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.583203 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-home\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.583361 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.583224 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-model-cache\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.583615 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.583518 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tmp-dir\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.583678 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.583642 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-model-cache\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.583678 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.583657 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-home\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.585584 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.585563 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-dshm\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.585956 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.585934 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tls-certs\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.592002 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.591976 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-kube-api-access-z8bjc\") pod \"sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.695201 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.695157 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:33.836722 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:33.836686 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8"] Jul 07 00:32:33.839738 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:32:33.839711 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2b6b9ee_608b_43cd_9062_9b4166d08bf7.slice/crio-76d20943e4b09bae96b4fb92d0fccee199998e8d6f56a7933d4e639664cb7fca WatchSource:0}: Error finding container 76d20943e4b09bae96b4fb92d0fccee199998e8d6f56a7933d4e639664cb7fca: Status 404 returned error can't find the container with id 76d20943e4b09bae96b4fb92d0fccee199998e8d6f56a7933d4e639664cb7fca Jul 07 00:32:34.557796 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:34.557760 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" event={"ID":"f2b6b9ee-608b-43cd-9062-9b4166d08bf7","Type":"ContainerStarted","Data":"e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688"} Jul 07 00:32:34.557796 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:34.557801 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" event={"ID":"f2b6b9ee-608b-43cd-9062-9b4166d08bf7","Type":"ContainerStarted","Data":"76d20943e4b09bae96b4fb92d0fccee199998e8d6f56a7933d4e639664cb7fca"} Jul 07 00:32:34.576949 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:34.576899 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" podStartSLOduration=1.576883065 podStartE2EDuration="1.576883065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:32:34.573970138 +0000 UTC m=+2192.519555512" watchObservedRunningTime="2026-07-07 00:32:34.576883065 +0000 UTC m=+2192.522468452" Jul 07 00:32:37.546040 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:37.546004 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:32:43.696152 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:43.696109 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:43.696740 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:43.696196 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:43.709173 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:43.709143 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:44.621102 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:44.621066 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:32:56.241668 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:32:56.241629 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:33:14.245710 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:14.245674 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:33:19.387734 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:19.387698 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp"] Jul 07 00:33:19.408148 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:19.408100 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:33:20.262461 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:20.262428 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8"] Jul 07 00:33:20.280362 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:20.280324 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:33:30.950251 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:30.950165 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:33:34.408979 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.408914 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" podUID="f040b410-0c36-4d2f-af28-25e19e62660f" containerName="main" containerID="cri-o://589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3" gracePeriod=45 Jul 07 00:33:34.664614 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.664526 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:33:34.740023 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.739975 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-tmp-dir\") pod \"f040b410-0c36-4d2f-af28-25e19e62660f\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " Jul 07 00:33:34.740023 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740028 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-model-cache\") pod \"f040b410-0c36-4d2f-af28-25e19e62660f\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " Jul 07 00:33:34.740262 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740051 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f040b410-0c36-4d2f-af28-25e19e62660f-tls-certs\") pod \"f040b410-0c36-4d2f-af28-25e19e62660f\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " Jul 07 00:33:34.740262 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740074 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-dshm\") pod \"f040b410-0c36-4d2f-af28-25e19e62660f\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " Jul 07 00:33:34.740262 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740122 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-home\") pod \"f040b410-0c36-4d2f-af28-25e19e62660f\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " Jul 07 00:33:34.740262 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740199 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2bd5\" (UniqueName: \"kubernetes.io/projected/f040b410-0c36-4d2f-af28-25e19e62660f-kube-api-access-r2bd5\") pod \"f040b410-0c36-4d2f-af28-25e19e62660f\" (UID: \"f040b410-0c36-4d2f-af28-25e19e62660f\") " Jul 07 00:33:34.740475 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740278 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f040b410-0c36-4d2f-af28-25e19e62660f" (UID: "f040b410-0c36-4d2f-af28-25e19e62660f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:33:34.740475 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740320 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f040b410-0c36-4d2f-af28-25e19e62660f" (UID: "f040b410-0c36-4d2f-af28-25e19e62660f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:33:34.740607 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740472 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-home" (OuterVolumeSpecName: "home") pod "f040b410-0c36-4d2f-af28-25e19e62660f" (UID: "f040b410-0c36-4d2f-af28-25e19e62660f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:33:34.740607 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740521 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-tmp-dir\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:34.740607 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.740569 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-model-cache\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:34.742406 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.742368 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-dshm" (OuterVolumeSpecName: "dshm") pod "f040b410-0c36-4d2f-af28-25e19e62660f" (UID: "f040b410-0c36-4d2f-af28-25e19e62660f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:33:34.742406 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.742385 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f040b410-0c36-4d2f-af28-25e19e62660f-kube-api-access-r2bd5" (OuterVolumeSpecName: "kube-api-access-r2bd5") pod "f040b410-0c36-4d2f-af28-25e19e62660f" (UID: "f040b410-0c36-4d2f-af28-25e19e62660f"). InnerVolumeSpecName "kube-api-access-r2bd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:33:34.742601 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.742405 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f040b410-0c36-4d2f-af28-25e19e62660f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f040b410-0c36-4d2f-af28-25e19e62660f" (UID: "f040b410-0c36-4d2f-af28-25e19e62660f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:33:34.841983 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.841944 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2bd5\" (UniqueName: \"kubernetes.io/projected/f040b410-0c36-4d2f-af28-25e19e62660f-kube-api-access-r2bd5\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:34.841983 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.841975 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f040b410-0c36-4d2f-af28-25e19e62660f-tls-certs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:34.841983 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.841986 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-dshm\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:34.841983 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.841994 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f040b410-0c36-4d2f-af28-25e19e62660f-home\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:34.873415 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.873380 2563 generic.go:356] "Generic (PLEG): container finished" podID="f040b410-0c36-4d2f-af28-25e19e62660f" containerID="589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3" exitCode=0 Jul 07 00:33:34.873606 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.873429 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" Jul 07 00:33:34.873606 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.873473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" event={"ID":"f040b410-0c36-4d2f-af28-25e19e62660f","Type":"ContainerDied","Data":"589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3"} Jul 07 00:33:34.873606 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.873520 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp" event={"ID":"f040b410-0c36-4d2f-af28-25e19e62660f","Type":"ContainerDied","Data":"a302844f3f435f69316cb04ea59d516a456d13399f3e50bbd7d9a6d037df8f55"} Jul 07 00:33:34.873606 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.873568 2563 scope.go:117] "RemoveContainer" containerID="589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3" Jul 07 00:33:34.883267 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.883241 2563 scope.go:117] "RemoveContainer" containerID="589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3" Jul 07 00:33:34.883592 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:33:34.883572 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3\": container with ID starting with 589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3 not found: ID does not exist" containerID="589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3" Jul 07 00:33:34.883646 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.883603 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3"} err="failed to get container status \"589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3\": rpc error: code = NotFound desc = could not find container \"589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3\": container with ID starting with 589a33935815b28b46314a81ba543d5c3f957d5998aa0b5809fa765a365d04d3 not found: ID does not exist" Jul 07 00:33:34.896179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.896143 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp"] Jul 07 00:33:34.899891 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:34.899861 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/sub-test-model-8714ac-kserve-749fb4bf99-hhsqp"] Jul 07 00:33:35.283821 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.283772 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" podUID="f2b6b9ee-608b-43cd-9062-9b4166d08bf7" containerName="main" containerID="cri-o://e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688" gracePeriod=45 Jul 07 00:33:35.535380 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.535308 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:33:35.649415 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649377 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-home\") pod \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " Jul 07 00:33:35.649415 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649417 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-dshm\") pod \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " Jul 07 00:33:35.649672 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649441 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tls-certs\") pod \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " Jul 07 00:33:35.649672 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649504 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tmp-dir\") pod \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " Jul 07 00:33:35.649672 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649528 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-model-cache\") pod \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " Jul 07 00:33:35.649672 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649618 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-kube-api-access-z8bjc\") pod \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\" (UID: \"f2b6b9ee-608b-43cd-9062-9b4166d08bf7\") " Jul 07 00:33:35.649894 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649710 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-home" (OuterVolumeSpecName: "home") pod "f2b6b9ee-608b-43cd-9062-9b4166d08bf7" (UID: "f2b6b9ee-608b-43cd-9062-9b4166d08bf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:33:35.649894 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649794 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2b6b9ee-608b-43cd-9062-9b4166d08bf7" (UID: "f2b6b9ee-608b-43cd-9062-9b4166d08bf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:33:35.649894 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649859 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2b6b9ee-608b-43cd-9062-9b4166d08bf7" (UID: "f2b6b9ee-608b-43cd-9062-9b4166d08bf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:33:35.650031 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649972 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tmp-dir\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:35.650031 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649987 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-model-cache\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:35.650031 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.649996 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-home\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:35.651802 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.651778 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-dshm" (OuterVolumeSpecName: "dshm") pod "f2b6b9ee-608b-43cd-9062-9b4166d08bf7" (UID: "f2b6b9ee-608b-43cd-9062-9b4166d08bf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:33:35.651802 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.651784 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2b6b9ee-608b-43cd-9062-9b4166d08bf7" (UID: "f2b6b9ee-608b-43cd-9062-9b4166d08bf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:33:35.651952 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.651807 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-kube-api-access-z8bjc" (OuterVolumeSpecName: "kube-api-access-z8bjc") pod "f2b6b9ee-608b-43cd-9062-9b4166d08bf7" (UID: "f2b6b9ee-608b-43cd-9062-9b4166d08bf7"). InnerVolumeSpecName "kube-api-access-z8bjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:33:35.751413 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.751370 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-kube-api-access-z8bjc\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:35.751413 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.751405 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-dshm\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:35.751413 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.751415 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b6b9ee-608b-43cd-9062-9b4166d08bf7-tls-certs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:33:35.880773 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.880675 2563 generic.go:356] "Generic (PLEG): container finished" podID="f2b6b9ee-608b-43cd-9062-9b4166d08bf7" containerID="e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688" exitCode=0 Jul 07 00:33:35.880773 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.880720 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" Jul 07 00:33:35.880773 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.880763 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" event={"ID":"f2b6b9ee-608b-43cd-9062-9b4166d08bf7","Type":"ContainerDied","Data":"e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688"} Jul 07 00:33:35.881015 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.880804 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8" event={"ID":"f2b6b9ee-608b-43cd-9062-9b4166d08bf7","Type":"ContainerDied","Data":"76d20943e4b09bae96b4fb92d0fccee199998e8d6f56a7933d4e639664cb7fca"} Jul 07 00:33:35.881015 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.880823 2563 scope.go:117] "RemoveContainer" containerID="e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688" Jul 07 00:33:35.890774 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.890753 2563 scope.go:117] "RemoveContainer" containerID="e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688" Jul 07 00:33:35.891114 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:33:35.891094 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688\": container with ID starting with e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688 not found: ID does not exist" containerID="e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688" Jul 07 00:33:35.891174 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.891124 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688"} err="failed to get container status \"e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688\": rpc error: code = NotFound desc = could not find container \"e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688\": container with ID starting with e09974880f6ad7dee35192f76fb53c31cc4407b9c69d1aa108ddcca0d6eee688 not found: ID does not exist" Jul 07 00:33:35.903874 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.903808 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8"] Jul 07 00:33:35.906251 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:35.906226 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/sub-test-model-4e3cf5-kserve-58cbb58c79-hxjv8"] Jul 07 00:33:36.581080 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:36.581042 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f040b410-0c36-4d2f-af28-25e19e62660f" path="/var/lib/kubelet/pods/f040b410-0c36-4d2f-af28-25e19e62660f/volumes" Jul 07 00:33:36.581599 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:36.581432 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b6b9ee-608b-43cd-9062-9b4166d08bf7" path="/var/lib/kubelet/pods/f2b6b9ee-608b-43cd-9062-9b4166d08bf7/volumes" Jul 07 00:33:39.559546 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.559499 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7"] Jul 07 00:33:39.560038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.559865 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b6b9ee-608b-43cd-9062-9b4166d08bf7" containerName="main" Jul 07 00:33:39.560038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.559877 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b6b9ee-608b-43cd-9062-9b4166d08bf7" containerName="main" Jul 07 00:33:39.560038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.559897 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f040b410-0c36-4d2f-af28-25e19e62660f" containerName="main" Jul 07 00:33:39.560038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.559902 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f040b410-0c36-4d2f-af28-25e19e62660f" containerName="main" Jul 07 00:33:39.560038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.559959 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b6b9ee-608b-43cd-9062-9b4166d08bf7" containerName="main" Jul 07 00:33:39.560038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.559968 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f040b410-0c36-4d2f-af28-25e19e62660f" containerName="main" Jul 07 00:33:39.564812 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.564786 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.567515 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.567489 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"openshift-service-ca.crt\"" Jul 07 00:33:39.568685 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.568647 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"default-dockercfg-ngv9s\"" Jul 07 00:33:39.568855 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.568714 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"rate-test-model-8714ac-kserve-self-signed-certs\"" Jul 07 00:33:39.568855 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.568769 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8714ac\"/\"kube-root-ca.crt\"" Jul 07 00:33:39.572394 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.572368 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7"] Jul 07 00:33:39.588126 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.588093 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:33:39.684736 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.684691 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9bs\" (UniqueName: \"kubernetes.io/projected/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-kube-api-access-dw9bs\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.684736 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.684743 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-home\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.685015 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.684778 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-dshm\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.685015 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.684829 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tmp-dir\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.685015 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.684848 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tls-certs\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.685015 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.684871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-model-cache\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.785945 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.785901 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9bs\" (UniqueName: \"kubernetes.io/projected/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-kube-api-access-dw9bs\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.785945 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.785953 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-home\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.786223 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.785989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-dshm\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.786223 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.786017 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tmp-dir\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.786223 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.786041 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tls-certs\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.786223 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.786076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-model-cache\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.786507 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.786480 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tmp-dir\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.786583 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.786493 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-home\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.786623 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.786579 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-model-cache\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.788551 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.788518 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-dshm\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.788817 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.788798 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tls-certs\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.794196 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.794165 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9bs\" (UniqueName: \"kubernetes.io/projected/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-kube-api-access-dw9bs\") pod \"rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:39.875762 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:39.875660 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:40.018311 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:40.018287 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7"] Jul 07 00:33:40.020750 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:33:40.020723 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc459a0b9_3a8f_4afb_9051_92a8e111ce9f.slice/crio-6b4fa60f0903897371756f6bae8fa7ee842b7285a7b5cb51fbd515faf4ecc15d WatchSource:0}: Error finding container 6b4fa60f0903897371756f6bae8fa7ee842b7285a7b5cb51fbd515faf4ecc15d: Status 404 returned error can't find the container with id 6b4fa60f0903897371756f6bae8fa7ee842b7285a7b5cb51fbd515faf4ecc15d Jul 07 00:33:40.909428 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:40.909384 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" event={"ID":"c459a0b9-3a8f-4afb-9051-92a8e111ce9f","Type":"ContainerStarted","Data":"40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1"} Jul 07 00:33:40.909428 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:40.909429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" event={"ID":"c459a0b9-3a8f-4afb-9051-92a8e111ce9f","Type":"ContainerStarted","Data":"6b4fa60f0903897371756f6bae8fa7ee842b7285a7b5cb51fbd515faf4ecc15d"} Jul 07 00:33:40.926443 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:40.926377 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" podStartSLOduration=1.9263588569999999 podStartE2EDuration="1.926358857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:33:40.925500662 +0000 UTC m=+2258.871086034" watchObservedRunningTime="2026-07-07 00:33:40.926358857 +0000 UTC m=+2258.871944229" Jul 07 00:33:44.049390 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:44.049353 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:33:49.875955 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:49.875907 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:49.876406 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:49.875975 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:49.888702 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:49.888669 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:49.966088 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:49.966056 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:33:52.723045 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:52.723005 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:33:52.823608 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:33:52.823567 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:07.457277 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.457243 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729"] Jul 07 00:34:07.460507 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.460486 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.463183 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.463161 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"openshift-service-ca.crt\"" Jul 07 00:34:07.463321 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.463187 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"rate-test-model-4e3cf5-kserve-self-signed-certs\"" Jul 07 00:34:07.464190 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.464166 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"kube-root-ca.crt\"" Jul 07 00:34:07.464315 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.464191 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4e3cf5\"/\"default-dockercfg-dv9q9\"" Jul 07 00:34:07.469418 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.469393 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729"] Jul 07 00:34:07.485970 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.485931 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:07.543294 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.543258 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n477\" (UniqueName: \"kubernetes.io/projected/1c1037f0-d670-4e5a-ba95-24f3a7047af5-kube-api-access-6n477\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.543467 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.543362 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-home\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.543467 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.543391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-dshm\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.543467 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.543419 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tls-certs\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.543467 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.543445 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-model-cache\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.543626 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.543481 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tmp-dir\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.644936 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.644896 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-home\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.644936 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.644941 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-dshm\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.645179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.644972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tls-certs\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.645179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.645011 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-model-cache\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.645179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.645057 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tmp-dir\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.645179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.645103 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n477\" (UniqueName: \"kubernetes.io/projected/1c1037f0-d670-4e5a-ba95-24f3a7047af5-kube-api-access-6n477\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.645391 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.645370 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-home\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.645477 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.645422 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-model-cache\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.645647 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.645557 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tmp-dir\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.647624 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.647594 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-dshm\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.647853 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.647832 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tls-certs\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.652959 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.652937 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n477\" (UniqueName: \"kubernetes.io/projected/1c1037f0-d670-4e5a-ba95-24f3a7047af5-kube-api-access-6n477\") pod \"rate-test-model-4e3cf5-kserve-769b559755-pf729\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.772320 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.772228 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:07.913940 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:07.913866 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729"] Jul 07 00:34:07.915791 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:34:07.915758 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c1037f0_d670_4e5a_ba95_24f3a7047af5.slice/crio-59ee7b78935b53f0a919493bb17607d4a8111a7d8819cfb0ec4628cabc008903 WatchSource:0}: Error finding container 59ee7b78935b53f0a919493bb17607d4a8111a7d8819cfb0ec4628cabc008903: Status 404 returned error can't find the container with id 59ee7b78935b53f0a919493bb17607d4a8111a7d8819cfb0ec4628cabc008903 Jul 07 00:34:08.047981 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:08.047886 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" event={"ID":"1c1037f0-d670-4e5a-ba95-24f3a7047af5","Type":"ContainerStarted","Data":"38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2"} Jul 07 00:34:08.047981 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:08.047932 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" event={"ID":"1c1037f0-d670-4e5a-ba95-24f3a7047af5","Type":"ContainerStarted","Data":"59ee7b78935b53f0a919493bb17607d4a8111a7d8819cfb0ec4628cabc008903"} Jul 07 00:34:08.065782 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:08.065714 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" podStartSLOduration=1.0656937499999999 podStartE2EDuration="1.06569375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:34:08.065299203 +0000 UTC m=+2286.010884576" watchObservedRunningTime="2026-07-07 00:34:08.06569375 +0000 UTC m=+2286.011279121" Jul 07 00:34:09.113463 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:09.113431 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:09.214216 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:09.214181 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:11.947212 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:11.947179 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:34:16.784189 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:16.784146 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:16.891706 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:16.891665 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:17.773041 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:17.772999 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:17.773370 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:17.773091 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:17.785920 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:17.785892 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:18.110853 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:18.110772 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:34:29.349789 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:29.349755 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:34:41.945566 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:41.945511 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:42.938811 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:42.938772 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:34:49.636213 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:49.636179 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7"] Jul 07 00:34:49.651189 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:49.651150 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:50.541627 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:50.541592 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729"] Jul 07 00:34:50.582109 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:50.582075 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:51.591341 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:51.591291 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:51.994888 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:51.994848 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:52.106631 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:52.106583 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:52.200203 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:52.200161 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:52.370852 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:52.370757 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:34:55.244920 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:55.244883 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:34:57.061686 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:34:57.061656 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:04.657423 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.657375 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" podUID="c459a0b9-3a8f-4afb-9051-92a8e111ce9f" containerName="main" containerID="cri-o://40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1" gracePeriod=45 Jul 07 00:35:04.903014 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.902985 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:35:04.957004 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.956920 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-dshm\") pod \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " Jul 07 00:35:04.957004 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.956977 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tmp-dir\") pod \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " Jul 07 00:35:04.957212 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957009 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-model-cache\") pod \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " Jul 07 00:35:04.957212 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957063 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw9bs\" (UniqueName: \"kubernetes.io/projected/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-kube-api-access-dw9bs\") pod \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " Jul 07 00:35:04.957212 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957106 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-home\") pod \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " Jul 07 00:35:04.957212 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957175 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tls-certs\") pod \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\" (UID: \"c459a0b9-3a8f-4afb-9051-92a8e111ce9f\") " Jul 07 00:35:04.957427 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957321 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c459a0b9-3a8f-4afb-9051-92a8e111ce9f" (UID: "c459a0b9-3a8f-4afb-9051-92a8e111ce9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:04.957427 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957327 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c459a0b9-3a8f-4afb-9051-92a8e111ce9f" (UID: "c459a0b9-3a8f-4afb-9051-92a8e111ce9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:04.957427 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957359 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-home" (OuterVolumeSpecName: "home") pod "c459a0b9-3a8f-4afb-9051-92a8e111ce9f" (UID: "c459a0b9-3a8f-4afb-9051-92a8e111ce9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:04.957599 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957501 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tmp-dir\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:04.957599 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957514 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-model-cache\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:04.957599 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.957525 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-home\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:04.959382 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.959356 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-kube-api-access-dw9bs" (OuterVolumeSpecName: "kube-api-access-dw9bs") pod "c459a0b9-3a8f-4afb-9051-92a8e111ce9f" (UID: "c459a0b9-3a8f-4afb-9051-92a8e111ce9f"). InnerVolumeSpecName "kube-api-access-dw9bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:35:04.959382 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.959371 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c459a0b9-3a8f-4afb-9051-92a8e111ce9f" (UID: "c459a0b9-3a8f-4afb-9051-92a8e111ce9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:35:04.959550 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:04.959355 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-dshm" (OuterVolumeSpecName: "dshm") pod "c459a0b9-3a8f-4afb-9051-92a8e111ce9f" (UID: "c459a0b9-3a8f-4afb-9051-92a8e111ce9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:05.058719 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.058683 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-tls-certs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:05.058719 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.058714 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-dshm\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:05.058719 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.058724 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9bs\" (UniqueName: \"kubernetes.io/projected/c459a0b9-3a8f-4afb-9051-92a8e111ce9f-kube-api-access-dw9bs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:05.341387 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.341292 2563 generic.go:356] "Generic (PLEG): container finished" podID="c459a0b9-3a8f-4afb-9051-92a8e111ce9f" containerID="40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1" exitCode=0 Jul 07 00:35:05.341387 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.341336 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" Jul 07 00:35:05.341387 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.341350 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" event={"ID":"c459a0b9-3a8f-4afb-9051-92a8e111ce9f","Type":"ContainerDied","Data":"40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1"} Jul 07 00:35:05.341387 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.341383 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7" event={"ID":"c459a0b9-3a8f-4afb-9051-92a8e111ce9f","Type":"ContainerDied","Data":"6b4fa60f0903897371756f6bae8fa7ee842b7285a7b5cb51fbd515faf4ecc15d"} Jul 07 00:35:05.341712 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.341400 2563 scope.go:117] "RemoveContainer" containerID="40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1" Jul 07 00:35:05.351401 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.351379 2563 scope.go:117] "RemoveContainer" containerID="40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1" Jul 07 00:35:05.351756 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:35:05.351737 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1\": container with ID starting with 40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1 not found: ID does not exist" containerID="40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1" Jul 07 00:35:05.351867 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.351768 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1"} err="failed to get container status \"40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1\": rpc error: code = NotFound desc = could not find container \"40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1\": container with ID starting with 40fde67564ffc14f11a811c233a6eb6b7bf06c67a0720c79cb9c4eb4936797c1 not found: ID does not exist" Jul 07 00:35:05.363386 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.363352 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7"] Jul 07 00:35:05.367544 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.367503 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8714ac/rate-test-model-8714ac-kserve-7d6b7b99cf-svxv7"] Jul 07 00:35:05.562921 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.562885 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" podUID="1c1037f0-d670-4e5a-ba95-24f3a7047af5" containerName="main" containerID="cri-o://38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2" gracePeriod=45 Jul 07 00:35:05.818883 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.818851 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:35:05.864875 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.864837 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tls-certs\") pod \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " Jul 07 00:35:05.865038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.864927 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-model-cache\") pod \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " Jul 07 00:35:05.865038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.864952 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n477\" (UniqueName: \"kubernetes.io/projected/1c1037f0-d670-4e5a-ba95-24f3a7047af5-kube-api-access-6n477\") pod \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " Jul 07 00:35:05.865038 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.864971 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-dshm\") pod \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " Jul 07 00:35:05.865168 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.865042 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tmp-dir\") pod \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " Jul 07 00:35:05.865168 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.865070 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-home\") pod \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\" (UID: \"1c1037f0-d670-4e5a-ba95-24f3a7047af5\") " Jul 07 00:35:05.865288 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.865253 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c1037f0-d670-4e5a-ba95-24f3a7047af5" (UID: "1c1037f0-d670-4e5a-ba95-24f3a7047af5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:05.865448 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.865425 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c1037f0-d670-4e5a-ba95-24f3a7047af5" (UID: "1c1037f0-d670-4e5a-ba95-24f3a7047af5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:05.865567 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.865514 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-home" (OuterVolumeSpecName: "home") pod "1c1037f0-d670-4e5a-ba95-24f3a7047af5" (UID: "1c1037f0-d670-4e5a-ba95-24f3a7047af5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:05.867267 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.867244 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-dshm" (OuterVolumeSpecName: "dshm") pod "1c1037f0-d670-4e5a-ba95-24f3a7047af5" (UID: "1c1037f0-d670-4e5a-ba95-24f3a7047af5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:35:05.867267 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.867247 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c1037f0-d670-4e5a-ba95-24f3a7047af5" (UID: "1c1037f0-d670-4e5a-ba95-24f3a7047af5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:35:05.867416 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.867281 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1037f0-d670-4e5a-ba95-24f3a7047af5-kube-api-access-6n477" (OuterVolumeSpecName: "kube-api-access-6n477") pod "1c1037f0-d670-4e5a-ba95-24f3a7047af5" (UID: "1c1037f0-d670-4e5a-ba95-24f3a7047af5"). InnerVolumeSpecName "kube-api-access-6n477". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:35:05.966281 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.966183 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tls-certs\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:05.966281 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.966216 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-model-cache\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:05.966281 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.966227 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n477\" (UniqueName: \"kubernetes.io/projected/1c1037f0-d670-4e5a-ba95-24f3a7047af5-kube-api-access-6n477\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:05.966281 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.966236 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-dshm\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:05.966281 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.966245 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-tmp-dir\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:05.966281 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:05.966253 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c1037f0-d670-4e5a-ba95-24f3a7047af5-home\") on node \"ip-10-0-133-6.ec2.internal\" DevicePath \"\"" Jul 07 00:35:06.348930 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.348845 2563 generic.go:356] "Generic (PLEG): container finished" podID="1c1037f0-d670-4e5a-ba95-24f3a7047af5" containerID="38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2" exitCode=0 Jul 07 00:35:06.348930 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.348894 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" Jul 07 00:35:06.349144 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.348932 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" event={"ID":"1c1037f0-d670-4e5a-ba95-24f3a7047af5","Type":"ContainerDied","Data":"38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2"} Jul 07 00:35:06.349144 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.348981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729" event={"ID":"1c1037f0-d670-4e5a-ba95-24f3a7047af5","Type":"ContainerDied","Data":"59ee7b78935b53f0a919493bb17607d4a8111a7d8819cfb0ec4628cabc008903"} Jul 07 00:35:06.349144 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.349005 2563 scope.go:117] "RemoveContainer" containerID="38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2" Jul 07 00:35:06.358495 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.358472 2563 scope.go:117] "RemoveContainer" containerID="38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2" Jul 07 00:35:06.358810 ip-10-0-133-6 kubenswrapper[2563]: E0707 00:35:06.358791 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2\": container with ID starting with 38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2 not found: ID does not exist" containerID="38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2" Jul 07 00:35:06.358860 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.358819 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2"} err="failed to get container status \"38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2\": rpc error: code = NotFound desc = could not find container \"38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2\": container with ID starting with 38b0c7db404d48b101ff16b571a25e6ff99cc4ec516e88ff01c5b0d6cd1b5ed2 not found: ID does not exist" Jul 07 00:35:06.372428 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.372392 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729"] Jul 07 00:35:06.378190 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.378160 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4e3cf5/rate-test-model-4e3cf5-kserve-769b559755-pf729"] Jul 07 00:35:06.577786 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.577750 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c1037f0-d670-4e5a-ba95-24f3a7047af5" path="/var/lib/kubelet/pods/1c1037f0-d670-4e5a-ba95-24f3a7047af5/volumes" Jul 07 00:35:06.578234 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:06.578213 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c459a0b9-3a8f-4afb-9051-92a8e111ce9f" path="/var/lib/kubelet/pods/c459a0b9-3a8f-4afb-9051-92a8e111ce9f/volumes" Jul 07 00:35:33.954685 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:33.954644 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:34.221181 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:34.221079 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:34.451343 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:34.451301 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:34.612318 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:34.612230 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:35.017104 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:35.017070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:35.121444 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:35.121406 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:35.217815 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:35.217777 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:35.363366 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:35.363270 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:37.849350 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:37.849313 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nkt97"] Jul 07 00:35:40.911054 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:40.911014 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:40.976691 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:40.976656 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:41.178790 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:41.178712 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:41.419862 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:41.419820 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:46.592022 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:46.591983 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-btcv5_6e56473a-90eb-48a1-b8eb-63d574565722/manager/0.log" Jul 07 00:35:47.701849 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:47.701816 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7bd8fcf-qft7d_e7a3bbe0-438d-4f5a-84de-51b268f2d2bf/manager/0.log" Jul 07 00:35:47.710653 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:47.710620 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7bd8fcf-qft7d_e7a3bbe0-438d-4f5a-84de-51b268f2d2bf/copy-manifests/0.log" Jul 07 00:35:48.628173 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:48.628147 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5_6136da15-a8e7-42a8-a387-1f4c7cd74ff7/util/0.log" Jul 07 00:35:48.634908 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:48.634876 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5_6136da15-a8e7-42a8-a387-1f4c7cd74ff7/pull/0.log" Jul 07 00:35:48.641126 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:48.641099 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5_6136da15-a8e7-42a8-a387-1f4c7cd74ff7/extract/0.log" Jul 07 00:35:48.754403 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:48.754362 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs_ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376/util/0.log" Jul 07 00:35:48.761435 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:48.761408 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs_ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376/pull/0.log" Jul 07 00:35:48.768626 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:48.768605 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs_ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376/extract/0.log" Jul 07 00:35:48.882928 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:48.882829 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw_50991f3d-a1cf-4c51-a611-1f9c136f7749/util/0.log" Jul 07 00:35:48.889671 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:48.889641 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw_50991f3d-a1cf-4c51-a611-1f9c136f7749/pull/0.log" Jul 07 00:35:48.896729 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:48.896707 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw_50991f3d-a1cf-4c51-a611-1f9c136f7749/extract/0.log" Jul 07 00:35:49.008280 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:49.008253 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k_07956dfa-fedf-47cb-8211-9768035b3c00/pull/0.log" Jul 07 00:35:49.014644 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:49.014614 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k_07956dfa-fedf-47cb-8211-9768035b3c00/extract/0.log" Jul 07 00:35:49.020707 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:49.020682 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k_07956dfa-fedf-47cb-8211-9768035b3c00/util/0.log" Jul 07 00:35:49.476778 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:49.476747 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dlpj6_fb58690a-1cef-476c-923b-5ee24e057af7/kuadrant-console-plugin/0.log" Jul 07 00:35:49.829809 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:49.829732 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nkt97_bf7d7d6f-c676-404c-8c19-cf1063d41a97/limitador/0.log" Jul 07 00:35:50.532409 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:50.532378 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5dbc58cc-7t9k9_4de610c6-bd2a-4956-8026-25c882cfa4f2/kube-auth-proxy/0.log" Jul 07 00:35:51.714597 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:51.714565 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-54xwq_78466abb-2729-4d5f-8501-a53f11132356/main/0.log" Jul 07 00:35:51.941427 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:51.941389 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2r4rg_2dea6624-519f-4aad-872d-9cb50ed978ff/main/0.log" Jul 07 00:35:52.051275 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:52.051197 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-b8ns8_8cdf799b-ea6f-45ee-bacb-bc86e019ddee/main/0.log" Jul 07 00:35:55.906412 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:55.906367 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:55.964212 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:55.964177 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:35:58.869744 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:58.869711 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4qtnq_f1d07f90-5abf-453f-bc2a-920be5e548a8/global-pull-secret-syncer/0.log" Jul 07 00:35:59.019099 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:59.019060 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" Jul 07 00:35:59.134325 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:35:59.134243 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-6.ec2.internal_d45b6b823d2fa18cb79d62947f7a1859/haproxy/0.log" Jul 07 00:36:03.827705 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:03.827668 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5_6136da15-a8e7-42a8-a387-1f4c7cd74ff7/extract/0.log" Jul 07 00:36:03.855571 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:03.855514 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5_6136da15-a8e7-42a8-a387-1f4c7cd74ff7/util/0.log" Jul 07 00:36:03.879406 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:03.879374 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dpgj5_6136da15-a8e7-42a8-a387-1f4c7cd74ff7/pull/0.log" Jul 07 00:36:03.908229 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:03.908191 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs_ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376/extract/0.log" Jul 07 00:36:03.931993 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:03.931959 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs_ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376/util/0.log" Jul 07 00:36:03.956118 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:03.956082 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bljfs_ed9c918f-0b3e-45ef-a1c7-c6e39bbd0376/pull/0.log" Jul 07 00:36:03.983030 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:03.983002 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw_50991f3d-a1cf-4c51-a611-1f9c136f7749/extract/0.log" Jul 07 00:36:04.004300 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:04.004273 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw_50991f3d-a1cf-4c51-a611-1f9c136f7749/util/0.log" Jul 07 00:36:04.026839 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:04.026803 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73f5zsw_50991f3d-a1cf-4c51-a611-1f9c136f7749/pull/0.log" Jul 07 00:36:04.054446 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:04.054421 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k_07956dfa-fedf-47cb-8211-9768035b3c00/extract/0.log" Jul 07 00:36:04.077051 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:04.077022 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k_07956dfa-fedf-47cb-8211-9768035b3c00/util/0.log" Jul 07 00:36:04.100289 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:04.100203 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r2l5k_07956dfa-fedf-47cb-8211-9768035b3c00/pull/0.log" Jul 07 00:36:04.582743 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:04.582716 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dlpj6_fb58690a-1cef-476c-923b-5ee24e057af7/kuadrant-console-plugin/0.log" Jul 07 00:36:04.809042 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:04.809016 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nkt97_bf7d7d6f-c676-404c-8c19-cf1063d41a97/limitador/0.log" Jul 07 00:36:06.750179 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:06.750148 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-24dwp_0cd70d5d-727b-46d9-aeb5-8b611c620946/kube-state-metrics/0.log" Jul 07 00:36:06.768360 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:06.768336 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-24dwp_0cd70d5d-727b-46d9-aeb5-8b611c620946/kube-rbac-proxy-main/0.log" Jul 07 00:36:06.790775 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:06.790746 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-24dwp_0cd70d5d-727b-46d9-aeb5-8b611c620946/kube-rbac-proxy-self/0.log" Jul 07 00:36:06.818752 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:06.818722 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-767fc4b76f-vjxx9_368845f5-c1ee-42d0-a210-b083f94eaf04/metrics-server/0.log" Jul 07 00:36:07.031471 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.031388 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kw5nb_c0aabe90-1c40-4edb-8585-b5de80fdeeee/node-exporter/0.log" Jul 07 00:36:07.068081 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.068052 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kw5nb_c0aabe90-1c40-4edb-8585-b5de80fdeeee/kube-rbac-proxy/0.log" Jul 07 00:36:07.087507 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.087480 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kw5nb_c0aabe90-1c40-4edb-8585-b5de80fdeeee/init-textfile/0.log" Jul 07 00:36:07.449162 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.449133 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c6bc74d8b-2kxkt_c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0/telemeter-client/0.log" Jul 07 00:36:07.469933 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.469900 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c6bc74d8b-2kxkt_c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0/reload/0.log" Jul 07 00:36:07.491646 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.491618 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c6bc74d8b-2kxkt_c5c3d8d9-9b19-49e5-9d52-7c13465a5ed0/kube-rbac-proxy/0.log" Jul 07 00:36:07.526298 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.526263 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bc589d866-pgm7r_23cbc791-6397-465e-b1a9-f294ca405fc1/thanos-query/0.log" Jul 07 00:36:07.549765 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.549741 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bc589d866-pgm7r_23cbc791-6397-465e-b1a9-f294ca405fc1/kube-rbac-proxy-web/0.log" Jul 07 00:36:07.569108 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.569078 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bc589d866-pgm7r_23cbc791-6397-465e-b1a9-f294ca405fc1/kube-rbac-proxy/0.log" Jul 07 00:36:07.592305 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.592277 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bc589d866-pgm7r_23cbc791-6397-465e-b1a9-f294ca405fc1/prom-label-proxy/0.log" Jul 07 00:36:07.615238 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.615212 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bc589d866-pgm7r_23cbc791-6397-465e-b1a9-f294ca405fc1/kube-rbac-proxy-rules/0.log" Jul 07 00:36:07.634926 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:07.634899 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bc589d866-pgm7r_23cbc791-6397-465e-b1a9-f294ca405fc1/kube-rbac-proxy-metrics/0.log" Jul 07 00:36:08.898498 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.898457 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh"] Jul 07 00:36:08.899008 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.898824 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c459a0b9-3a8f-4afb-9051-92a8e111ce9f" containerName="main" Jul 07 00:36:08.899008 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.898836 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c459a0b9-3a8f-4afb-9051-92a8e111ce9f" containerName="main" Jul 07 00:36:08.899008 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.898850 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1037f0-d670-4e5a-ba95-24f3a7047af5" containerName="main" Jul 07 00:36:08.899008 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.898855 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1037f0-d670-4e5a-ba95-24f3a7047af5" containerName="main" Jul 07 00:36:08.899008 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.898921 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1037f0-d670-4e5a-ba95-24f3a7047af5" containerName="main" Jul 07 00:36:08.899008 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.898932 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c459a0b9-3a8f-4afb-9051-92a8e111ce9f" containerName="main" Jul 07 00:36:08.901961 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.901939 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:08.904421 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.904399 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5sg4w\"/\"default-dockercfg-h2ksb\"" Jul 07 00:36:08.904505 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.904399 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5sg4w\"/\"openshift-service-ca.crt\"" Jul 07 00:36:08.905432 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.905415 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5sg4w\"/\"kube-root-ca.crt\"" Jul 07 00:36:08.912705 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:08.912684 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh"] Jul 07 00:36:09.017147 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.017110 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmrr\" (UniqueName: \"kubernetes.io/projected/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-kube-api-access-vfmrr\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.017347 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.017197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-podres\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.017347 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.017267 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-sys\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.017347 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.017318 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-proc\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.017474 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.017359 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-lib-modules\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.118638 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.118603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-proc\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.118814 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.118650 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-lib-modules\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.118814 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.118690 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmrr\" (UniqueName: \"kubernetes.io/projected/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-kube-api-access-vfmrr\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.118814 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.118716 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-podres\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.118814 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.118733 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-proc\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.118814 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.118749 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-sys\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.118814 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.118805 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-sys\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.119060 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.118869 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-podres\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.119060 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.118875 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-lib-modules\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.127013 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.126992 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmrr\" (UniqueName: \"kubernetes.io/projected/fb3b46e0-4c6a-4e16-b82f-7e92996b8531-kube-api-access-vfmrr\") pod \"perf-node-gather-daemonset-nz2nh\" (UID: \"fb3b46e0-4c6a-4e16-b82f-7e92996b8531\") " pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.212608 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.212503 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.349120 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.349095 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh"] Jul 07 00:36:09.351065 ip-10-0-133-6 kubenswrapper[2563]: W0707 00:36:09.351035 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfb3b46e0_4c6a_4e16_b82f_7e92996b8531.slice/crio-8b8d7bfef8e41a5824076c319097222cc61c78e613b657501de6077c07ea13e7 WatchSource:0}: Error finding container 8b8d7bfef8e41a5824076c319097222cc61c78e613b657501de6077c07ea13e7: Status 404 returned error can't find the container with id 8b8d7bfef8e41a5824076c319097222cc61c78e613b657501de6077c07ea13e7 Jul 07 00:36:09.666838 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.666796 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" event={"ID":"fb3b46e0-4c6a-4e16-b82f-7e92996b8531","Type":"ContainerStarted","Data":"1fcf07a65a27798b5eb4a30158397b5d30fea63bf414f63e4a5de76d95fde90e"} Jul 07 00:36:09.667036 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.666862 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:09.667036 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.666887 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" event={"ID":"fb3b46e0-4c6a-4e16-b82f-7e92996b8531","Type":"ContainerStarted","Data":"8b8d7bfef8e41a5824076c319097222cc61c78e613b657501de6077c07ea13e7"} Jul 07 00:36:09.682648 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:09.682595 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" podStartSLOduration=1.682577547 podStartE2EDuration="1.682577547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 00:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:36:09.681725465 +0000 UTC m=+2407.627310837" watchObservedRunningTime="2026-07-07 00:36:09.682577547 +0000 UTC m=+2407.628162917" Jul 07 00:36:11.647353 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:11.647327 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kps6x_b75bb351-dbc1-47d5-bbd8-39714250e069/dns/0.log" Jul 07 00:36:11.667454 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:11.667430 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kps6x_b75bb351-dbc1-47d5-bbd8-39714250e069/kube-rbac-proxy/0.log" Jul 07 00:36:11.756298 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:11.756264 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5tmhb_c4cb04e9-3bdd-4c7b-8b9f-8063c3605454/dns-node-resolver/0.log" Jul 07 00:36:12.286338 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:12.286298 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29723040-z8bg8_e38b8c49-6cc0-4234-8b07-4b957681ca29/image-pruner/0.log" Jul 07 00:36:12.332858 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:12.332822 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84b64754fb-tdx8n_553125df-b156-4489-8db0-934fba7fa1d6/registry/0.log" Jul 07 00:36:12.354597 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:12.354570 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jrnh_aa00bed5-0833-4872-99cb-7c9d933d9323/node-ca/0.log" Jul 07 00:36:13.444734 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:13.444702 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5dbc58cc-7t9k9_4de610c6-bd2a-4956-8026-25c882cfa4f2/kube-auth-proxy/0.log" Jul 07 00:36:14.290815 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:14.290780 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2zd2z_22b1ce4e-ded4-4fc5-9089-48f68aebb1f0/serve-healthcheck-canary/0.log" Jul 07 00:36:14.912404 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:14.912371 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cd577_61f39557-e0fa-416c-ab65-0c3ef436d755/kube-rbac-proxy/0.log" Jul 07 00:36:14.931970 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:14.931942 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cd577_61f39557-e0fa-416c-ab65-0c3ef436d755/exporter/0.log" Jul 07 00:36:14.953442 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:14.953409 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cd577_61f39557-e0fa-416c-ab65-0c3ef436d755/extractor/0.log" Jul 07 00:36:15.682355 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:15.682327 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5sg4w/perf-node-gather-daemonset-nz2nh" Jul 07 00:36:17.257790 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:17.257752 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:36:17.331834 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:17.331801 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:36:17.410646 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:17.410611 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-btcv5_6e56473a-90eb-48a1-b8eb-63d574565722/manager/0.log" Jul 07 00:36:17.930781 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:17.930746 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7bd8fcf-qft7d_e7a3bbe0-438d-4f5a-84de-51b268f2d2bf/manager/0.log" Jul 07 00:36:17.950885 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:17.950860 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7bd8fcf-qft7d_e7a3bbe0-438d-4f5a-84de-51b268f2d2bf/copy-manifests/0.log" Jul 07 00:36:19.295314 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:19.295281 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5dbc58cc-7t9k9_4de610c6-bd2a-4956-8026-25c882cfa4f2/kube-auth-proxy/0.log" Jul 07 00:36:19.414992 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:19.414961 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6647df896d-w4zh8_eb2d6a38-3149-427d-aa83-beb38726ea43/manager/0.log" Jul 07 00:36:20.851142 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:20.851104 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5dbc58cc-7t9k9_4de610c6-bd2a-4956-8026-25c882cfa4f2/kube-auth-proxy/0.log" Jul 07 00:36:21.236490 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:21.236451 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:36:21.288989 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:21.288958 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rfjl_99c62bce-e3c3-4979-ac9e-5c3d187870df/konnectivity-agent/0.log" limit=1024 Jul 07 00:36:25.475395 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:25.475366 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82nrx_ff834cf2-cbcd-4722-9c4e-597f34f52036/kube-multus-additional-cni-plugins/0.log" Jul 07 00:36:25.496720 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:25.496691 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82nrx_ff834cf2-cbcd-4722-9c4e-597f34f52036/egress-router-binary-copy/0.log" Jul 07 00:36:25.520234 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:25.520206 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82nrx_ff834cf2-cbcd-4722-9c4e-597f34f52036/cni-plugins/0.log" Jul 07 00:36:25.542488 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:25.542455 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82nrx_ff834cf2-cbcd-4722-9c4e-597f34f52036/bond-cni-plugin/0.log" Jul 07 00:36:25.565610 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:25.565585 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82nrx_ff834cf2-cbcd-4722-9c4e-597f34f52036/routeoverride-cni/0.log" Jul 07 00:36:25.587400 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:25.587371 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82nrx_ff834cf2-cbcd-4722-9c4e-597f34f52036/whereabouts-cni-bincopy/0.log" Jul 07 00:36:25.608285 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:25.608251 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82nrx_ff834cf2-cbcd-4722-9c4e-597f34f52036/whereabouts-cni/0.log" Jul 07 00:36:25.995466 ip-10-0-133-6 kubenswrapper[2563]: I0707 00:36:25.995425 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jmpjk_02c4a893-1ad4-4389-81b7-9bb7ded935b6/kube-multus/0.log"