Jul 10 17:09:08.806813 ip-10-0-140-169 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 17:09:08.806827 ip-10-0-140-169 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 17:09:08.806834 ip-10-0-140-169 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 17:09:08.807144 ip-10-0-140-169 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 17:09:18.922856 ip-10-0-140-169 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 17:09:18.922871 ip-10-0-140-169 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ed6fee2b11a64087bc6440df106d6532 -- Jul 10 17:11:13.795968 ip-10-0-140-169 systemd[1]: Starting Kubernetes Kubelet... Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.279637 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284865 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284876 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284880 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284883 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284886 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284889 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284892 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:14.394913 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284895 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:14.359596 ip-10-0-140-169 systemd[1]: Started Kubernetes Kubelet. Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284898 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284901 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284904 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284908 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284911 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284913 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284916 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284919 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284921 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284926 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284930 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284932 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284935 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284938 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284941 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284944 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284947 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284949 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:14.396014 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284954 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284958 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284961 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284964 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284967 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284970 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284973 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284976 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284979 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284981 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284984 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284986 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284989 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284992 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284995 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.284997 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285000 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285003 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285006 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285008 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:14.396691 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285011 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285013 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285016 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285018 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285021 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285023 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285026 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285028 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285031 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285034 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285037 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285040 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285042 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285046 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285049 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285051 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285054 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285056 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285059 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:14.397411 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285062 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285064 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285067 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285069 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285072 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285074 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285077 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285079 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285082 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285085 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285087 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285090 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285092 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285095 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285097 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285100 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285102 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285105 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285107 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285111 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:14.398061 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285113 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285116 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285120 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285123 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285126 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285128 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285131 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285134 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285137 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285140 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285142 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285145 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285149 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285152 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285837 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285844 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285848 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285851 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285854 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:14.398636 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285857 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285860 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285863 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285866 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285868 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285871 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285874 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285877 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285880 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285882 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285885 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285888 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285890 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285893 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285896 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285899 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285902 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285905 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285908 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285910 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:14.399330 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285913 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285916 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285918 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285921 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285924 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285926 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285929 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285931 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285934 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285936 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285939 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285942 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285944 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285947 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285950 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285952 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285955 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285958 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285961 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:14.400006 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285964 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285966 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285971 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285973 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285976 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285979 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285981 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285983 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285986 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285988 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285991 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285994 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285997 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.285999 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286002 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286004 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286007 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286010 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286012 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286015 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286018 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:14.400620 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286020 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286023 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286025 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286028 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286031 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286034 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286040 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286045 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286048 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286050 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286053 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286056 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286058 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286062 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286065 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286068 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286071 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286074 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286076 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:14.401363 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286080 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286084 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286087 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286091 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286094 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286096 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286099 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286102 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286104 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286108 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286112 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286115 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286117 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286120 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286123 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286488 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286493 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286497 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286500 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286504 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:14.402011 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286507 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286509 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286512 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286515 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286517 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286521 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286525 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286528 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286530 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286533 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286536 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286539 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286542 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286545 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286548 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286551 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286554 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286557 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286559 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:14.402598 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286562 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286564 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286569 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286571 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286574 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286576 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286579 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286582 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286584 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286587 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286589 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286592 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286596 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286599 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286601 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286604 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286607 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286609 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286613 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286616 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:14.403240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286619 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286621 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286624 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286626 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286629 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286631 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286634 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286637 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286639 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286642 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286645 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286648 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286650 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286653 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286656 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286659 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286661 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286664 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286666 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:14.404004 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286669 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286672 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286675 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286677 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286680 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286683 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286686 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286689 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286691 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286695 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286698 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286716 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286719 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286722 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286725 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286728 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286730 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286733 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286736 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286739 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:14.404561 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286742 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286744 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286747 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286750 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286755 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286758 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286762 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286766 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286769 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286772 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286774 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286777 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286780 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286782 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286785 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.286787 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286850 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286863 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286871 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286875 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286879 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 17:11:14.405857 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286882 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286886 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286890 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286894 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286896 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286900 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286903 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286906 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286909 2580 flags.go:64] FLAG: --cgroup-root="" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286912 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286915 2580 flags.go:64] FLAG: --client-ca-file="" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286918 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286921 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286924 2580 flags.go:64] FLAG: --cluster-domain="" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286927 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286931 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286934 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286937 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286942 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286945 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286948 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286951 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286954 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286957 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 17:11:14.406477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286960 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286963 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286966 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286970 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286975 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286978 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286980 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286983 2580 flags.go:64] FLAG: --enable-server="true" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286986 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286990 2580 flags.go:64] FLAG: --event-burst="100" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286993 2580 flags.go:64] FLAG: --event-qps="50" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286996 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.286999 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287002 2580 flags.go:64] FLAG: --eviction-hard="" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287005 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287008 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287011 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287014 2580 flags.go:64] FLAG: --eviction-soft="" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287016 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287019 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287022 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287025 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287028 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287030 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287034 2580 flags.go:64] FLAG: --feature-gates="" Jul 10 17:11:14.407161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287037 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287041 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287044 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287047 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287050 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287053 2580 flags.go:64] FLAG: --help="false" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287056 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-169.ec2.internal" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287060 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287062 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287065 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287069 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287073 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287076 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287079 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287082 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287085 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287088 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287091 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287094 2580 flags.go:64] FLAG: --kube-reserved="" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287096 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287100 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287103 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287105 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287108 2580 flags.go:64] FLAG: --lock-file="" Jul 10 17:11:14.408007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287111 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287114 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287117 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287122 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287125 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287128 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287130 2580 flags.go:64] FLAG: --logging-format="text" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287133 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287164 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287208 2580 flags.go:64] FLAG: --manifest-url="" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287214 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287220 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287224 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287229 2580 flags.go:64] FLAG: --max-pods="110" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287234 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287237 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287240 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287248 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287251 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287257 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287263 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287275 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287282 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287784 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 17:11:14.612054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287805 2580 flags.go:64] FLAG: --pod-cidr="" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287813 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287823 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287829 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287836 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287843 2580 flags.go:64] FLAG: --port="10250" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287849 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287862 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0da98733b4c8d05db" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287869 2580 flags.go:64] FLAG: --qos-reserved="" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287876 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287882 2580 flags.go:64] FLAG: --register-node="true" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287887 2580 flags.go:64] FLAG: --register-with-taints="" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287907 2580 flags.go:64] FLAG: --registry-burst="10" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287912 2580 flags.go:64] FLAG: --registry-qps="5" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287917 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287922 2580 flags.go:64] FLAG: --reserved-memory="" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287936 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287942 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287947 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287953 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287958 2580 flags.go:64] FLAG: --runonce="false" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287964 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287970 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287976 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287987 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 17:11:14.613385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287993 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.287999 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288005 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288010 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288015 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288021 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288026 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288033 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288044 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288049 2580 flags.go:64] FLAG: --system-cgroups="" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288054 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288072 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288086 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288092 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288101 2580 flags.go:64] FLAG: --tls-min-version="" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288139 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288149 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288155 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288162 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288167 2580 flags.go:64] FLAG: --v="2" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288175 2580 flags.go:64] FLAG: --version="false" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288182 2580 flags.go:64] FLAG: --vmodule="" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288224 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.288360 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288486 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:14.617747 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288491 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288494 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288497 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288500 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288504 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288507 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288509 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288512 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288515 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288521 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288525 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288529 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288531 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288534 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288537 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288540 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288543 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288546 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288549 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:14.618600 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288551 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288554 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288557 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288559 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288562 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288566 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288569 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288572 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288575 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288577 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288580 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288583 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288586 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288588 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288591 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288594 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288597 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288599 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288602 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:14.619234 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288605 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288609 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288612 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288617 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288620 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288622 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288625 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288628 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288631 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288633 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288636 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288639 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288641 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288644 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288646 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288649 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288652 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288654 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288657 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288659 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:14.619916 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288662 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288665 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288667 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288670 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288673 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288675 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288678 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288680 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288683 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288685 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288688 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288690 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288693 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288695 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288698 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288717 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288720 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288722 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288725 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288728 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:14.620500 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288731 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288734 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288737 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288740 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288742 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288747 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288750 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288753 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288755 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288758 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288761 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288764 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288767 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288769 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288772 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288774 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288777 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288780 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288782 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:14.621160 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.288785 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.289578 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.297027 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.297040 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.298086 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.298093 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298145 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298150 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298154 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298157 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298159 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298162 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298165 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298168 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298171 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:14.621717 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298173 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298176 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298178 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298182 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298184 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298187 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298190 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298192 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298195 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298197 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298200 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298202 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298205 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298208 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298210 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298213 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298216 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298219 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298223 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298226 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:14.622243 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298230 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298232 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298235 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298238 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298241 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298250 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298252 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298255 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298257 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298260 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298263 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298265 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298268 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298270 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298272 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298275 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298279 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298281 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298284 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298287 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:14.622845 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298289 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298292 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298296 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298299 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298303 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298306 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298309 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298312 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298316 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298319 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298322 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298326 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298328 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298331 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298334 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298337 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298339 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298343 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298346 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:14.623503 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298349 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298351 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298354 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298357 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298359 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298362 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298365 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298368 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298371 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298374 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298377 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298379 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298382 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298384 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298387 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298389 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298392 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298394 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298397 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:14.624171 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298399 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298402 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298404 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298407 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298409 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298411 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298414 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298417 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298419 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298421 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298424 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298428 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.298433 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298549 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298554 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:14.624882 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298557 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298560 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298563 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298566 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298569 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298572 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298574 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298577 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298580 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298583 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298586 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298589 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298594 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298597 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298599 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298602 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298605 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298609 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298612 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:14.625372 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298616 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298620 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298623 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298626 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298630 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298632 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298635 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298638 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298641 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298645 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298648 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298651 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298654 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298656 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298659 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298661 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298664 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298667 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298670 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298672 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:14.626010 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298675 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298677 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298680 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298683 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298685 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298689 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298692 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298694 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298697 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298699 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298724 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298727 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298729 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298732 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298735 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298737 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298740 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298742 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298745 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:14.873898 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298748 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298750 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298755 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298757 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298760 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298763 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298765 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298768 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298770 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298773 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298775 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298778 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298780 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298783 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298786 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298789 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298791 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298795 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298799 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298801 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:14.875119 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298804 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298807 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298809 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298812 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298815 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298817 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298820 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298823 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298825 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298828 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298830 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298833 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298835 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298838 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298841 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298844 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298846 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298849 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:14.875797 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:14.298852 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.298857 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.298948 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.301237 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.302991 2580 server.go:1019] "Starting client certificate rotation" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.303088 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.303125 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.305928 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.307019 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.318455 2580 log.go:25] "Validated CRI v1 runtime API" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.323569 2580 log.go:25] "Validated CRI v1 image API" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.324316 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 17:11:14.876429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.328748 2580 fs.go:135] Filesystem UUIDs: map[325e01fc-7bff-4535-987d-62ed90f3c588:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cc12251b-eb6a-43f6-b814-238dcfb7fe76:/dev/nvme0n1p4] Jul 10 17:11:14.876854 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.328762 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 17:11:14.877053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.334984 2580 manager.go:216] Machine: {Timestamp:2026-07-10 17:11:14.332337904 +0000 UTC m=+0.401534995 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099693 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22605fda92ab0e6157d4921ae807e6 SystemUUID:ec22605f-da92-ab0e-6157-d4921ae807e6 BootID:ed6fee2b-11a6-4087-bc64-40df106d6532 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:55:d1:ef:63:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:55:d1:ef:63:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:f3:44:1d:a6:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 17:11:14.877053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.335091 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 17:11:14.877053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.335163 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 17:11:14.877053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.335842 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:11:14.877053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.339476 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 17:11:14.877053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.339500 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-169.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 17:11:14.877053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.341278 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.341287 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.341294 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.341305 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.342621 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.346110 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.346231 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.346258 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.346266 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.346280 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.346289 2580 kubelet.go:398] "Adding apiserver pod source" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.346297 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.346666 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.346684 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.350898 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.352958 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.352985 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353113 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353126 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353132 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353137 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353145 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353150 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353788 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353795 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353804 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353809 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353817 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.353825 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.355525 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.355538 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.358617 2580 server.go:1268] "Started kubelet" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.358681 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.358726 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.358800 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.358975 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-169.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.360055 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.360056 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-169.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.362871 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ftnj5" Jul 10 17:11:14.877693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.364670 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.365307 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.365312 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.366619 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-169.ec2.internal.18c0fcdc05c61a1f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-169.ec2.internal,UID:ip-10-0-140-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-169.ec2.internal,},FirstTimestamp:2026-07-10 17:11:14.358589983 +0000 UTC m=+0.427787074,LastTimestamp:2026-07-10 17:11:14.358589983 +0000 UTC m=+0.427787074,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-169.ec2.internal,}" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.371638 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ftnj5" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.372149 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.372151 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.372230 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.372854 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.372854 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.372877 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.372964 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.372973 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.373247 2580 factory.go:153] Registering CRI-O factory Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.373263 2580 factory.go:223] Registration of the crio container factory successfully Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.373332 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.373340 2580 factory.go:55] Registering systemd factory Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.373345 2580 factory.go:223] Registration of the systemd container factory successfully Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.373363 2580 factory.go:103] Registering Raw factory Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.373374 2580 manager.go:1196] Started watching for new ooms in manager Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.373649 2580 manager.go:349] Starting recovery of all containers Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.383930 2580 manager.go:354] Recovery completed Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.389373 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.391590 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.391619 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.391631 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.392123 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.392136 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.392155 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.392338 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.393170 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.393196 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.393206 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.393371 2580 policy_none.go:49] "None policy: Start" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.393391 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.393410 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 17:11:14.878872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.394501 2580 policy_none.go:47] "Start" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.396639 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-169.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.396759 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.444914 2580 manager.go:337] "Starting Device Plugin manager" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.444945 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.444954 2580 server.go:87] "Starting device plugin registration server" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.445207 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.445219 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.445302 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.445385 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.445394 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.445966 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.446011 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.494344 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.495440 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.495476 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.495500 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.495539 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.499922 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.546884 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.548001 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.548032 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.548044 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.548071 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.557025 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.557046 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-169.ec2.internal\": node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.575846 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.596193 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal"] Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.596268 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.597217 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.597245 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.597257 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.599698 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.599835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.057702 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.599876 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.600449 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.600449 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.600482 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.600497 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.600482 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.600559 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.602361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.602386 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.603237 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.603260 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.603270 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.639589 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-169.ec2.internal\" not found" node="ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.644236 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-169.ec2.internal\" not found" node="ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.674303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b5f2664fb83b457695239877b264372c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"b5f2664fb83b457695239877b264372c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.674327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5f2664fb83b457695239877b264372c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"b5f2664fb83b457695239877b264372c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.674347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27bd9ee91ed58c4c427c2a52d0f7a6ad-config\") pod \"kube-apiserver-proxy-ip-10-0-140-169.ec2.internal\" (UID: \"27bd9ee91ed58c4c427c2a52d0f7a6ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.676548 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.775028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b5f2664fb83b457695239877b264372c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"b5f2664fb83b457695239877b264372c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.775055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5f2664fb83b457695239877b264372c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"b5f2664fb83b457695239877b264372c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.775073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27bd9ee91ed58c4c427c2a52d0f7a6ad-config\") pod \"kube-apiserver-proxy-ip-10-0-140-169.ec2.internal\" (UID: \"27bd9ee91ed58c4c427c2a52d0f7a6ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.059444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.775116 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27bd9ee91ed58c4c427c2a52d0f7a6ad-config\") pod \"kube-apiserver-proxy-ip-10-0-140-169.ec2.internal\" (UID: \"27bd9ee91ed58c4c427c2a52d0f7a6ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.060113 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.775129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b5f2664fb83b457695239877b264372c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"b5f2664fb83b457695239877b264372c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.060113 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.775146 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5f2664fb83b457695239877b264372c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"b5f2664fb83b457695239877b264372c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.060113 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.777308 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.060113 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.877962 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.060113 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.946349 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.060113 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:14.947530 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.060113 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:14.978660 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.303183 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:15.079774 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.303183 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:15.180795 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.303183 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.214133 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:15.303183 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:15.281452 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.305599 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.303217 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 17:11:15.305599 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.303356 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 17:11:15.305599 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.303389 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 17:11:15.519230 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.375960 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 17:06:14 +0000 UTC" deadline="2027-12-31 16:29:34.908747648 +0000 UTC" Jul 10 17:11:15.519230 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.375982 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12935h18m19.532768095s" Jul 10 17:11:15.519230 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:15.381582 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.519230 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:15.482521 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.692621 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:15.583507 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found" Jul 10 17:11:15.692621 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:15.639439 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bd9ee91ed58c4c427c2a52d0f7a6ad.slice/crio-722205ff70b82bb78c8c1ac77ff223a3059dcc7b4f3691946f4965a8c838f89c WatchSource:0}: Error finding container 722205ff70b82bb78c8c1ac77ff223a3059dcc7b4f3691946f4965a8c838f89c: Status 404 returned error can't find the container with id 722205ff70b82bb78c8c1ac77ff223a3059dcc7b4f3691946f4965a8c838f89c Jul 10 17:11:15.692621 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:15.639685 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f2664fb83b457695239877b264372c.slice/crio-5d41f4cb3a9ef3dc31c26caa88702acd3f883bb045ad707634f9e56aeca9b5e9 WatchSource:0}: Error finding container 5d41f4cb3a9ef3dc31c26caa88702acd3f883bb045ad707634f9e56aeca9b5e9: Status 404 returned error can't find the container with id 5d41f4cb3a9ef3dc31c26caa88702acd3f883bb045ad707634f9e56aeca9b5e9 Jul 10 17:11:15.692621 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.644792 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:11:15.692621 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.646155 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:15.692621 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.672952 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.692621 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.686282 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:11:15.692621 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.687653 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" Jul 10 17:11:15.696318 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.695997 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:11:15.908444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:15.889192 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.349949 2580 apiserver.go:52] "Watching apiserver" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.357283 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.358665 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.361756 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9ngr2","openshift-network-diagnostics/network-check-target-95wjr","openshift-ovn-kubernetes/ovnkube-node-jc2mr","kube-system/global-pull-secret-syncer-pw595","kube-system/konnectivity-agent-4874w","openshift-dns/node-resolver-n6672","openshift-image-registry/node-ca-69wk5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal","openshift-multus/multus-m5n45","openshift-multus/network-metrics-daemon-cks5v","openshift-network-operator/iptables-alerter-7xgxk","kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n","openshift-cluster-node-tuning-operator/tuned-clvg4"] Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.363525 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.365694 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.367578 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.367821 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.368195 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gld9\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.368413 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.368555 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.368739 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.369539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.369871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.369984 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.371290 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.371693 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ld7xh\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.372879 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.372963 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.373317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.374435 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.374665 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.374869 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.375227 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.376038 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mjgrt\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.376147 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 17:11:16.405692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.405193 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.405577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-etc-openvswitch\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-dbus\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-device-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-run-systemd\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-run-openvswitch\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgdpz\" (UniqueName: \"kubernetes.io/projected/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-kube-api-access-xgdpz\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-systemd-units\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-run-netns\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-node-log\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.406982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-cni-bin\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-ovn-node-metrics-cert\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.407594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-ovnkube-script-lib\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-cni-netd\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407102 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-ovnkube-config\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-kubelet-config\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a301d47-a92b-46d1-aaf9-08ecf8ffccd6-agent-certs\") pod \"konnectivity-agent-4874w\" (UID: \"9a301d47-a92b-46d1-aaf9-08ecf8ffccd6\") " pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xhb\" (UniqueName: \"kubernetes.io/projected/2667319a-4b88-41ab-bba5-44f2b50e4269-kube-api-access-b9xhb\") pod \"node-ca-69wk5\" (UID: \"2667319a-4b88-41ab-bba5-44f2b50e4269\") " pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-registration-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-kubelet\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2667319a-4b88-41ab-bba5-44f2b50e4269-host\") pod \"node-ca-69wk5\" (UID: \"2667319a-4b88-41ab-bba5-44f2b50e4269\") " pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-socket-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-etc-selinux\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvg4\" (UniqueName: \"kubernetes.io/projected/0b5dbde9-7175-4f6a-b833-2891f1286aeb-kube-api-access-vsvg4\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.408844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-slash\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-run-ovn\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-log-socket\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-env-overrides\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a301d47-a92b-46d1-aaf9-08ecf8ffccd6-konnectivity-ca\") pod \"konnectivity-agent-4874w\" (UID: \"9a301d47-a92b-46d1-aaf9-08ecf8ffccd6\") " pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-sys-fs\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2667319a-4b88-41ab-bba5-44f2b50e4269-serviceca\") pod \"node-ca-69wk5\" (UID: \"2667319a-4b88-41ab-bba5-44f2b50e4269\") " pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.407688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-var-lib-openvswitch\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.408331 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.408357 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.408629 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c7d8t\"" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.408775 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.408913 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 17:11:16.409692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.409230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.411624 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.409918 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 17:11:16.411624 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.409945 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 17:11:16.411624 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.410101 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 17:11:16.411624 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.410144 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 17:11:16.411624 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.410406 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xl2n8\"" Jul 10 17:11:16.411624 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.411009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:16.411624 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.411072 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:16.415330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.412408 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-79jhq\"" Jul 10 17:11:16.415330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.412452 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 17:11:16.415330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.413201 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 17:11:16.415330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.413321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:16.415330 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.413388 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:16.415330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.413494 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rlcnb\"" Jul 10 17:11:16.415330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.414908 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.419717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.417133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.419717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.418171 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:11:16.419717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.418371 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 17:11:16.419717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.418391 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5jp2b\"" Jul 10 17:11:16.419717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.418245 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 17:11:16.419717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.418580 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:11:16.423448 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.420243 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:11:16.423448 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.420436 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cnqnd\"" Jul 10 17:11:16.423448 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.420616 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 17:11:16.484324 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.428832 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zrpkl" Jul 10 17:11:16.484324 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.436565 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zrpkl" Jul 10 17:11:16.484324 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.483181 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.501633 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" event={"ID":"b5f2664fb83b457695239877b264372c","Type":"ContainerStarted","Data":"5d41f4cb3a9ef3dc31c26caa88702acd3f883bb045ad707634f9e56aeca9b5e9"} Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.503036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" event={"ID":"27bd9ee91ed58c4c427c2a52d0f7a6ad","Type":"ContainerStarted","Data":"722205ff70b82bb78c8c1ac77ff223a3059dcc7b4f3691946f4965a8c838f89c"} Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/671bb016-c27a-48ce-8aaa-408d5f05999f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-var-lib-kubelet\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a301d47-a92b-46d1-aaf9-08ecf8ffccd6-konnectivity-ca\") pod \"konnectivity-agent-4874w\" (UID: \"9a301d47-a92b-46d1-aaf9-08ecf8ffccd6\") " pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nsml\" (UniqueName: \"kubernetes.io/projected/078c83e5-04a5-474a-8a60-4a0b8aa280d2-kube-api-access-4nsml\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpbwz\" (UniqueName: \"kubernetes.io/projected/7e9c8570-8386-4aad-bbbb-54b00f6ad38b-kube-api-access-lpbwz\") pod \"iptables-alerter-7xgxk\" (UID: \"7e9c8570-8386-4aad-bbbb-54b00f6ad38b\") " pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508416 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-sysctl-d\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhj8\" (UniqueName: \"kubernetes.io/projected/4942f89d-529f-4889-82e4-0056933168ed-kube-api-access-5vhj8\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2667319a-4b88-41ab-bba5-44f2b50e4269-serviceca\") pod \"node-ca-69wk5\" (UID: \"2667319a-4b88-41ab-bba5-44f2b50e4269\") " pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4942f89d-529f-4889-82e4-0056933168ed-cni-binary-copy\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.547444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-tuned\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-var-lib-openvswitch\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-device-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-var-lib-openvswitch\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-modprobe-d\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-sysconfig\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk67d\" (UniqueName: \"kubernetes.io/projected/2f9594ea-c699-482c-914d-d8dd3803ea38-kube-api-access-tk67d\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lkh\" (UniqueName: \"kubernetes.io/projected/671bb016-c27a-48ce-8aaa-408d5f05999f-kube-api-access-75lkh\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-run-openvswitch\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508765 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a301d47-a92b-46d1-aaf9-08ecf8ffccd6-konnectivity-ca\") pod \"konnectivity-agent-4874w\" (UID: \"9a301d47-a92b-46d1-aaf9-08ecf8ffccd6\") " pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgdpz\" (UniqueName: \"kubernetes.io/projected/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-kube-api-access-xgdpz\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-run\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-os-release\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-multus-cni-dir\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.548556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-multus-socket-dir-parent\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-run-k8s-cni-cncf-io\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-hostroot\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.508977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-device-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-systemd-units\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-run-netns\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e9c8570-8386-4aad-bbbb-54b00f6ad38b-iptables-alerter-script\") pod \"iptables-alerter-7xgxk\" (UID: \"7e9c8570-8386-4aad-bbbb-54b00f6ad38b\") " pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-kubernetes\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-run-openvswitch\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-var-lib-cni-multus\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2667319a-4b88-41ab-bba5-44f2b50e4269-serviceca\") pod \"node-ca-69wk5\" (UID: \"2667319a-4b88-41ab-bba5-44f2b50e4269\") " pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-systemd-units\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509294 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.549185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-run-netns\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-run-multus-certs\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-cni-netd\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-ovnkube-config\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-cni-netd\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a301d47-a92b-46d1-aaf9-08ecf8ffccd6-agent-certs\") pod \"konnectivity-agent-4874w\" (UID: \"9a301d47-a92b-46d1-aaf9-08ecf8ffccd6\") " pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/671bb016-c27a-48ce-8aaa-408d5f05999f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-kubelet\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-kubelet\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-socket-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-etc-selinux\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvg4\" (UniqueName: \"kubernetes.io/projected/0b5dbde9-7175-4f6a-b833-2891f1286aeb-kube-api-access-vsvg4\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-sysctl-conf\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.550592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-host\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-etc-kubernetes\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e9ce868-b07d-4674-be8b-3f27b781f174-hosts-file\") pod \"node-resolver-n6672\" (UID: \"6e9ce868-b07d-4674-be8b-3f27b781f174\") " pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509875 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-cnibin\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/671bb016-c27a-48ce-8aaa-408d5f05999f-cni-binary-copy\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-multus-conf-dir\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-log-socket\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-env-overrides\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509961 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-sys-fs\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.509980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-sys-fs\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510461 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-ovnkube-config\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:16.551501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510525 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-log-socket\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-systemd\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.510554 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.510621 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret podName:b9842d85-1f4d-4c40-a1ff-2254160fdb9e nodeName:}" failed. No retries permitted until 2026-07-10 17:11:17.010597598 +0000 UTC m=+3.079794691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret") pod "global-pull-secret-syncer-pw595" (UID: "b9842d85-1f4d-4c40-a1ff-2254160fdb9e") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-var-lib-kubelet\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4942f89d-529f-4889-82e4-0056933168ed-multus-daemon-config\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510795 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f9594ea-c699-482c-914d-d8dd3803ea38-tmp\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-env-overrides\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e9ce868-b07d-4674-be8b-3f27b781f174-tmp-dir\") pod \"node-resolver-n6672\" (UID: \"6e9ce868-b07d-4674-be8b-3f27b781f174\") " pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-system-cni-dir\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-os-release\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-etc-selinux\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-etc-openvswitch\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-socket-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.553037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-dbus\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-etc-openvswitch\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.510990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-lib-modules\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-cnibin\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-run-netns\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-run-systemd\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-node-log\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-dbus\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-node-log\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-cni-bin\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-run-systemd\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-ovn-node-metrics-cert\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.553962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511208 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-cni-bin\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-ovnkube-script-lib\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xhb\" (UniqueName: \"kubernetes.io/projected/2667319a-4b88-41ab-bba5-44f2b50e4269-kube-api-access-b9xhb\") pod \"node-ca-69wk5\" (UID: \"2667319a-4b88-41ab-bba5-44f2b50e4269\") " pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-var-lib-cni-bin\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-kubelet-config\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-registration-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9c8570-8386-4aad-bbbb-54b00f6ad38b-host-slash\") pod \"iptables-alerter-7xgxk\" (UID: \"7e9c8570-8386-4aad-bbbb-54b00f6ad38b\") " pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2667319a-4b88-41ab-bba5-44f2b50e4269-host\") pod \"node-ca-69wk5\" (UID: \"2667319a-4b88-41ab-bba5-44f2b50e4269\") " pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-slash\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-run-ovn\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-sys\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm46r\" (UniqueName: \"kubernetes.io/projected/6e9ce868-b07d-4674-be8b-3f27b781f174-kube-api-access-cm46r\") pod \"node-resolver-n6672\" (UID: \"6e9ce868-b07d-4674-be8b-3f27b781f174\") " pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511772 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-kubelet-config\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:16.555033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-system-cni-dir\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b5dbde9-7175-4f6a-b833-2891f1286aeb-registration-dir\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.511975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-run-ovn\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.512009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2667319a-4b88-41ab-bba5-44f2b50e4269-host\") pod \"node-ca-69wk5\" (UID: \"2667319a-4b88-41ab-bba5-44f2b50e4269\") " pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.512036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-host-slash\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.512427 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-ovnkube-script-lib\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.514114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a301d47-a92b-46d1-aaf9-08ecf8ffccd6-agent-certs\") pod \"konnectivity-agent-4874w\" (UID: \"9a301d47-a92b-46d1-aaf9-08ecf8ffccd6\") " pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.515920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-ovn-node-metrics-cert\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.521250 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xhb\" (UniqueName: \"kubernetes.io/projected/2667319a-4b88-41ab-bba5-44f2b50e4269-kube-api-access-b9xhb\") pod \"node-ca-69wk5\" (UID: \"2667319a-4b88-41ab-bba5-44f2b50e4269\") " pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.522463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgdpz\" (UniqueName: \"kubernetes.io/projected/ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1-kube-api-access-xgdpz\") pod \"ovnkube-node-jc2mr\" (UID: \"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.556645 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.524889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvg4\" (UniqueName: \"kubernetes.io/projected/0b5dbde9-7175-4f6a-b833-2891f1286aeb-kube-api-access-vsvg4\") pod \"aws-ebs-csi-driver-node-pkb2n\" (UID: \"0b5dbde9-7175-4f6a-b833-2891f1286aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.612904 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.612871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-run-multus-certs\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613043 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.612921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/671bb016-c27a-48ce-8aaa-408d5f05999f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.613043 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.612957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:16.613043 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.612992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-sysctl-conf\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.613043 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-host\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.613043 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-run-multus-certs\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-etc-kubernetes\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e9ce868-b07d-4674-be8b-3f27b781f174-hosts-file\") pod \"node-resolver-n6672\" (UID: \"6e9ce868-b07d-4674-be8b-3f27b781f174\") " pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-cnibin\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/671bb016-c27a-48ce-8aaa-408d5f05999f-cni-binary-copy\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-multus-conf-dir\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-systemd\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-var-lib-kubelet\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4942f89d-529f-4889-82e4-0056933168ed-multus-daemon-config\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e9ce868-b07d-4674-be8b-3f27b781f174-hosts-file\") pod \"node-resolver-n6672\" (UID: \"6e9ce868-b07d-4674-be8b-3f27b781f174\") " pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f9594ea-c699-482c-914d-d8dd3803ea38-tmp\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e9ce868-b07d-4674-be8b-3f27b781f174-tmp-dir\") pod \"node-resolver-n6672\" (UID: \"6e9ce868-b07d-4674-be8b-3f27b781f174\") " pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.613345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-etc-kubernetes\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-system-cni-dir\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-os-release\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-systemd\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-lib-modules\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613450 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-cnibin\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-cnibin\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613478 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-sysctl-conf\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-run-netns\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-var-lib-cni-bin\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-os-release\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9c8570-8386-4aad-bbbb-54b00f6ad38b-host-slash\") pod \"iptables-alerter-7xgxk\" (UID: \"7e9c8570-8386-4aad-bbbb-54b00f6ad38b\") " pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-sys\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm46r\" (UniqueName: \"kubernetes.io/projected/6e9ce868-b07d-4674-be8b-3f27b781f174-kube-api-access-cm46r\") pod \"node-resolver-n6672\" (UID: \"6e9ce868-b07d-4674-be8b-3f27b781f174\") " pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.613993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-system-cni-dir\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/671bb016-c27a-48ce-8aaa-408d5f05999f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-var-lib-kubelet\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nsml\" (UniqueName: \"kubernetes.io/projected/078c83e5-04a5-474a-8a60-4a0b8aa280d2-kube-api-access-4nsml\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbwz\" (UniqueName: \"kubernetes.io/projected/7e9c8570-8386-4aad-bbbb-54b00f6ad38b-kube-api-access-lpbwz\") pod \"iptables-alerter-7xgxk\" (UID: \"7e9c8570-8386-4aad-bbbb-54b00f6ad38b\") " pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-sysctl-d\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e9ce868-b07d-4674-be8b-3f27b781f174-tmp-dir\") pod \"node-resolver-n6672\" (UID: \"6e9ce868-b07d-4674-be8b-3f27b781f174\") " pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhj8\" (UniqueName: \"kubernetes.io/projected/4942f89d-529f-4889-82e4-0056933168ed-kube-api-access-5vhj8\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4942f89d-529f-4889-82e4-0056933168ed-cni-binary-copy\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614000 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-var-lib-kubelet\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.613339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-host\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614028 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/671bb016-c27a-48ce-8aaa-408d5f05999f-cni-binary-copy\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-var-lib-kubelet\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614115 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/671bb016-c27a-48ce-8aaa-408d5f05999f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.614685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-tuned\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-multus-conf-dir\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-sys\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614193 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-system-cni-dir\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-lib-modules\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-cnibin\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-modprobe-d\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.614319 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614359 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-modprobe-d\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.614390 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs podName:078c83e5-04a5-474a-8a60-4a0b8aa280d2 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:17.114372371 +0000 UTC m=+3.183569456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs") pod "network-metrics-daemon-cks5v" (UID: "078c83e5-04a5-474a-8a60-4a0b8aa280d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-sysconfig\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk67d\" (UniqueName: \"kubernetes.io/projected/2f9594ea-c699-482c-914d-d8dd3803ea38-kube-api-access-tk67d\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-sysctl-d\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.615488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lkh\" (UniqueName: \"kubernetes.io/projected/671bb016-c27a-48ce-8aaa-408d5f05999f-kube-api-access-75lkh\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-run\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/671bb016-c27a-48ce-8aaa-408d5f05999f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-os-release\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-var-lib-cni-bin\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-system-cni-dir\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614744 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-run-netns\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9c8570-8386-4aad-bbbb-54b00f6ad38b-host-slash\") pod \"iptables-alerter-7xgxk\" (UID: \"7e9c8570-8386-4aad-bbbb-54b00f6ad38b\") " pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-multus-cni-dir\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-multus-socket-dir-parent\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/671bb016-c27a-48ce-8aaa-408d5f05999f-os-release\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-run-k8s-cni-cncf-io\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-multus-cni-dir\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-run-k8s-cni-cncf-io\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-hostroot\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-run\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e9c8570-8386-4aad-bbbb-54b00f6ad38b-iptables-alerter-script\") pod \"iptables-alerter-7xgxk\" (UID: \"7e9c8570-8386-4aad-bbbb-54b00f6ad38b\") " pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-sysconfig\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.615006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-kubernetes\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.615047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-var-lib-cni-multus\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.615186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-host-var-lib-cni-multus\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.614969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-multus-socket-dir-parent\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.615273 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4942f89d-529f-4889-82e4-0056933168ed-hostroot\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.615332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-kubernetes\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.615366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4942f89d-529f-4889-82e4-0056933168ed-cni-binary-copy\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.615962 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e9c8570-8386-4aad-bbbb-54b00f6ad38b-iptables-alerter-script\") pod \"iptables-alerter-7xgxk\" (UID: \"7e9c8570-8386-4aad-bbbb-54b00f6ad38b\") " pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.615998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4942f89d-529f-4889-82e4-0056933168ed-multus-daemon-config\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.616945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.616594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f9594ea-c699-482c-914d-d8dd3803ea38-tmp\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.617900 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.617859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f9594ea-c699-482c-914d-d8dd3803ea38-etc-tuned\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.624150 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.623908 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:16.624150 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.623934 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:16.624150 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.623950 2580 projected.go:196] Error preparing data for projected volume kube-api-access-4smwb for pod openshift-network-diagnostics/network-check-target-95wjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:16.624150 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:16.624010 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb podName:9507138b-069a-4e78-8029-41513a50465d nodeName:}" failed. No retries permitted until 2026-07-10 17:11:17.123992063 +0000 UTC m=+3.193189164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4smwb" (UniqueName: "kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb") pod "network-check-target-95wjr" (UID: "9507138b-069a-4e78-8029-41513a50465d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:16.626358 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.626329 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm46r\" (UniqueName: \"kubernetes.io/projected/6e9ce868-b07d-4674-be8b-3f27b781f174-kube-api-access-cm46r\") pod \"node-resolver-n6672\" (UID: \"6e9ce868-b07d-4674-be8b-3f27b781f174\") " pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.627418 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.627351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lkh\" (UniqueName: \"kubernetes.io/projected/671bb016-c27a-48ce-8aaa-408d5f05999f-kube-api-access-75lkh\") pod \"multus-additional-cni-plugins-9ngr2\" (UID: \"671bb016-c27a-48ce-8aaa-408d5f05999f\") " pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.627418 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.627359 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk67d\" (UniqueName: \"kubernetes.io/projected/2f9594ea-c699-482c-914d-d8dd3803ea38-kube-api-access-tk67d\") pod \"tuned-clvg4\" (UID: \"2f9594ea-c699-482c-914d-d8dd3803ea38\") " pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:16.627666 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.627413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nsml\" (UniqueName: \"kubernetes.io/projected/078c83e5-04a5-474a-8a60-4a0b8aa280d2-kube-api-access-4nsml\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:16.628182 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.628137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhj8\" (UniqueName: \"kubernetes.io/projected/4942f89d-529f-4889-82e4-0056933168ed-kube-api-access-5vhj8\") pod \"multus-m5n45\" (UID: \"4942f89d-529f-4889-82e4-0056933168ed\") " pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.628969 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.628940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbwz\" (UniqueName: \"kubernetes.io/projected/7e9c8570-8386-4aad-bbbb-54b00f6ad38b-kube-api-access-lpbwz\") pod \"iptables-alerter-7xgxk\" (UID: \"7e9c8570-8386-4aad-bbbb-54b00f6ad38b\") " pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.680386 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.680304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-69wk5" Jul 10 17:11:16.692322 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:16.692264 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2667319a_4b88_41ab_bba5_44f2b50e4269.slice/crio-32fefc54295a7478f6694ffad640fd0068bd0a3667333b9efae487569b5c5737 WatchSource:0}: Error finding container 32fefc54295a7478f6694ffad640fd0068bd0a3667333b9efae487569b5c5737: Status 404 returned error can't find the container with id 32fefc54295a7478f6694ffad640fd0068bd0a3667333b9efae487569b5c5737 Jul 10 17:11:16.712088 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.712063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" Jul 10 17:11:16.726527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.726502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:16.730633 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.730614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:16.739267 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.739240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m5n45" Jul 10 17:11:16.749993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.749972 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n6672" Jul 10 17:11:16.765220 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.765199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" Jul 10 17:11:16.898323 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.898288 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xgxk" Jul 10 17:11:16.919050 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:16.919026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clvg4" Jul 10 17:11:17.017514 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:17.017416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:17.017674 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:17.017589 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:17.017674 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:17.017658 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret podName:b9842d85-1f4d-4c40-a1ff-2254160fdb9e nodeName:}" failed. No retries permitted until 2026-07-10 17:11:18.01763927 +0000 UTC m=+4.086836352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret") pod "global-pull-secret-syncer-pw595" (UID: "b9842d85-1f4d-4c40-a1ff-2254160fdb9e") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:17.117911 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:17.117870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:17.118105 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:17.118040 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:17.118174 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:17.118110 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs podName:078c83e5-04a5-474a-8a60-4a0b8aa280d2 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:18.118092825 +0000 UTC m=+4.187289907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs") pod "network-metrics-daemon-cks5v" (UID: "078c83e5-04a5-474a-8a60-4a0b8aa280d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:17.149212 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:17.149180 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:17.219306 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:17.219255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:17.219492 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:17.219432 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:17.219492 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:17.219455 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:17.219492 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:17.219467 2580 projected.go:196] Error preparing data for projected volume kube-api-access-4smwb for pod openshift-network-diagnostics/network-check-target-95wjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:17.219670 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:17.219534 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb podName:9507138b-069a-4e78-8029-41513a50465d nodeName:}" failed. No retries permitted until 2026-07-10 17:11:18.21951439 +0000 UTC m=+4.288711469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4smwb" (UniqueName: "kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb") pod "network-check-target-95wjr" (UID: "9507138b-069a-4e78-8029-41513a50465d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:17.439746 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:17.439609 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:06:16 +0000 UTC" deadline="2028-02-12 03:04:27.630396498 +0000 UTC" Jul 10 17:11:17.439746 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:17.439684 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13953h53m10.190717694s" Jul 10 17:11:17.505752 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:17.505703 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-69wk5" event={"ID":"2667319a-4b88-41ab-bba5-44f2b50e4269","Type":"ContainerStarted","Data":"32fefc54295a7478f6694ffad640fd0068bd0a3667333b9efae487569b5c5737"} Jul 10 17:11:17.756230 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:17.756055 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671bb016_c27a_48ce_8aaa_408d5f05999f.slice/crio-0a8d54bbdd17db8ab1d6ac73b0fb8c61868d595d9cbfd57cfe06c6d3f29ba16f WatchSource:0}: Error finding container 0a8d54bbdd17db8ab1d6ac73b0fb8c61868d595d9cbfd57cfe06c6d3f29ba16f: Status 404 returned error can't find the container with id 0a8d54bbdd17db8ab1d6ac73b0fb8c61868d595d9cbfd57cfe06c6d3f29ba16f Jul 10 17:11:17.758086 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:17.757664 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a301d47_a92b_46d1_aaf9_08ecf8ffccd6.slice/crio-d3e4fb47f52c7887506e9acaba87406c2dfa02c04a1883343c46783652e2d06d WatchSource:0}: Error finding container d3e4fb47f52c7887506e9acaba87406c2dfa02c04a1883343c46783652e2d06d: Status 404 returned error can't find the container with id d3e4fb47f52c7887506e9acaba87406c2dfa02c04a1883343c46783652e2d06d Jul 10 17:11:17.758547 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:17.758520 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5dbde9_7175_4f6a_b833_2891f1286aeb.slice/crio-3a6890997b580ed393208df3a483598c351f12dbdf26f930ab2ffd1d60f796c8 WatchSource:0}: Error finding container 3a6890997b580ed393208df3a483598c351f12dbdf26f930ab2ffd1d60f796c8: Status 404 returned error can't find the container with id 3a6890997b580ed393208df3a483598c351f12dbdf26f930ab2ffd1d60f796c8 Jul 10 17:11:17.760557 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:17.760533 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9594ea_c699_482c_914d_d8dd3803ea38.slice/crio-4ecf3c9ab78031f71239010689e5e819994c7fe45f6c78b9e351eb21850ce61a WatchSource:0}: Error finding container 4ecf3c9ab78031f71239010689e5e819994c7fe45f6c78b9e351eb21850ce61a: Status 404 returned error can't find the container with id 4ecf3c9ab78031f71239010689e5e819994c7fe45f6c78b9e351eb21850ce61a Jul 10 17:11:17.762210 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:17.762077 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9c8570_8386_4aad_bbbb_54b00f6ad38b.slice/crio-bb6cf74c367f9c828bfe25f270857a708cea30790d2143dc63fbd550c97575dd WatchSource:0}: Error finding container bb6cf74c367f9c828bfe25f270857a708cea30790d2143dc63fbd550c97575dd: Status 404 returned error can't find the container with id bb6cf74c367f9c828bfe25f270857a708cea30790d2143dc63fbd550c97575dd Jul 10 17:11:17.763240 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:11:17.763117 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4942f89d_529f_4889_82e4_0056933168ed.slice/crio-9a7a359ddd33e0e154a18823ed234f139b3d5ec5b82c56aa303847966929bd49 WatchSource:0}: Error finding container 9a7a359ddd33e0e154a18823ed234f139b3d5ec5b82c56aa303847966929bd49: Status 404 returned error can't find the container with id 9a7a359ddd33e0e154a18823ed234f139b3d5ec5b82c56aa303847966929bd49 Jul 10 17:11:18.027433 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.027411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:18.027571 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.027553 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:18.027620 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.027614 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret podName:b9842d85-1f4d-4c40-a1ff-2254160fdb9e nodeName:}" failed. No retries permitted until 2026-07-10 17:11:20.027594636 +0000 UTC m=+6.096791714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret") pod "global-pull-secret-syncer-pw595" (UID: "b9842d85-1f4d-4c40-a1ff-2254160fdb9e") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:18.128957 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.128748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:18.128957 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.128895 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:18.129168 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.129001 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs podName:078c83e5-04a5-474a-8a60-4a0b8aa280d2 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:20.12898157 +0000 UTC m=+6.198178648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs") pod "network-metrics-daemon-cks5v" (UID: "078c83e5-04a5-474a-8a60-4a0b8aa280d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:18.229592 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.229559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:18.229792 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.229775 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:18.229872 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.229825 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:18.229872 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.229839 2580 projected.go:196] Error preparing data for projected volume kube-api-access-4smwb for pod openshift-network-diagnostics/network-check-target-95wjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:18.229973 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.229892 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb podName:9507138b-069a-4e78-8029-41513a50465d nodeName:}" failed. No retries permitted until 2026-07-10 17:11:20.229874495 +0000 UTC m=+6.299071583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4smwb" (UniqueName: "kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb") pod "network-check-target-95wjr" (UID: "9507138b-069a-4e78-8029-41513a50465d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:18.440242 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.440133 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:06:16 +0000 UTC" deadline="2028-03-06 20:21:31.286294288 +0000 UTC" Jul 10 17:11:18.440242 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.440194 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14523h10m12.846104714s" Jul 10 17:11:18.500564 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.499641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:18.500564 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.499831 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:18.500564 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.500274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:18.500564 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.500377 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:18.500564 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.500441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:18.500564 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:18.500513 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:18.515907 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.515096 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5f2664fb83b457695239877b264372c" containerID="fc90603f61a9d0735f27be541ccc75ec9cb7189a00dcff33ac2fcb189d03b99f" exitCode=0 Jul 10 17:11:18.515907 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.515679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" event={"ID":"b5f2664fb83b457695239877b264372c","Type":"ContainerDied","Data":"fc90603f61a9d0735f27be541ccc75ec9cb7189a00dcff33ac2fcb189d03b99f"} Jul 10 17:11:18.538954 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.538895 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clvg4" event={"ID":"2f9594ea-c699-482c-914d-d8dd3803ea38","Type":"ContainerStarted","Data":"4ecf3c9ab78031f71239010689e5e819994c7fe45f6c78b9e351eb21850ce61a"} Jul 10 17:11:18.547736 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.547655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xgxk" event={"ID":"7e9c8570-8386-4aad-bbbb-54b00f6ad38b","Type":"ContainerStarted","Data":"bb6cf74c367f9c828bfe25f270857a708cea30790d2143dc63fbd550c97575dd"} Jul 10 17:11:18.558599 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.558541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4874w" event={"ID":"9a301d47-a92b-46d1-aaf9-08ecf8ffccd6","Type":"ContainerStarted","Data":"d3e4fb47f52c7887506e9acaba87406c2dfa02c04a1883343c46783652e2d06d"} Jul 10 17:11:18.567019 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.566938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" event={"ID":"671bb016-c27a-48ce-8aaa-408d5f05999f","Type":"ContainerStarted","Data":"0a8d54bbdd17db8ab1d6ac73b0fb8c61868d595d9cbfd57cfe06c6d3f29ba16f"} Jul 10 17:11:18.579761 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.579666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n6672" event={"ID":"6e9ce868-b07d-4674-be8b-3f27b781f174","Type":"ContainerStarted","Data":"5dfdac1d6242eff1c983af21a83f053cd9ee6754d0b635f719a08bf3e4a525ff"} Jul 10 17:11:18.589194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.588556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" event={"ID":"27bd9ee91ed58c4c427c2a52d0f7a6ad","Type":"ContainerStarted","Data":"b4f8afa435edfd6ff96c3492be13bff79c1653a8f018bd949367047cb1ae8b78"} Jul 10 17:11:18.613240 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.613211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerStarted","Data":"d086d098d8661803a8dd9869a178640dddfb3d40619062ce56771e4b2a6410f9"} Jul 10 17:11:18.622078 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.622052 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m5n45" event={"ID":"4942f89d-529f-4889-82e4-0056933168ed","Type":"ContainerStarted","Data":"9a7a359ddd33e0e154a18823ed234f139b3d5ec5b82c56aa303847966929bd49"} Jul 10 17:11:18.633831 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:18.633738 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" event={"ID":"0b5dbde9-7175-4f6a-b833-2891f1286aeb","Type":"ContainerStarted","Data":"3a6890997b580ed393208df3a483598c351f12dbdf26f930ab2ffd1d60f796c8"} Jul 10 17:11:19.643317 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:19.643252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" event={"ID":"b5f2664fb83b457695239877b264372c","Type":"ContainerStarted","Data":"cb624b2d9415b9aa75012b4222b250f96a0abd5a082fa7d6a58f4f5a4f4f5163"} Jul 10 17:11:19.664038 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:19.662531 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" podStartSLOduration=4.662452075 podStartE2EDuration="4.662452075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:18.619609623 +0000 UTC m=+4.688806724" watchObservedRunningTime="2026-07-10 17:11:19.662452075 +0000 UTC m=+5.731649166" Jul 10 17:11:19.664038 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:19.663775 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" podStartSLOduration=4.663764861 podStartE2EDuration="4.663764861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:19.662056376 +0000 UTC m=+5.731253477" watchObservedRunningTime="2026-07-10 17:11:19.663764861 +0000 UTC m=+5.732961963" Jul 10 17:11:20.051374 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:20.051291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:20.051541 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.051466 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:20.051541 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.051527 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret podName:b9842d85-1f4d-4c40-a1ff-2254160fdb9e nodeName:}" failed. No retries permitted until 2026-07-10 17:11:24.05150843 +0000 UTC m=+10.120705523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret") pod "global-pull-secret-syncer-pw595" (UID: "b9842d85-1f4d-4c40-a1ff-2254160fdb9e") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:20.153016 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:20.152419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:20.153016 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.152609 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:20.153016 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.152671 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs podName:078c83e5-04a5-474a-8a60-4a0b8aa280d2 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:24.152650946 +0000 UTC m=+10.221848026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs") pod "network-metrics-daemon-cks5v" (UID: "078c83e5-04a5-474a-8a60-4a0b8aa280d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:20.254273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:20.253659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:20.254273 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.253861 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:20.254273 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.253886 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:20.254273 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.253900 2580 projected.go:196] Error preparing data for projected volume kube-api-access-4smwb for pod openshift-network-diagnostics/network-check-target-95wjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:20.254273 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.253958 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb podName:9507138b-069a-4e78-8029-41513a50465d nodeName:}" failed. No retries permitted until 2026-07-10 17:11:24.253939148 +0000 UTC m=+10.323136247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4smwb" (UniqueName: "kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb") pod "network-check-target-95wjr" (UID: "9507138b-069a-4e78-8029-41513a50465d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:20.496934 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:20.496821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:20.496934 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:20.496905 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:20.497152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:20.496969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:20.497152 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.497080 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:20.497562 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.497527 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:20.497682 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:20.497613 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:22.496261 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:22.496177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:22.496696 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:22.496362 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:22.496696 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:22.496438 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:22.496696 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:22.496531 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:22.496696 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:22.496582 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:22.496967 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:22.496729 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:24.085638 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:24.085594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:24.086132 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.085778 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:24.086132 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.085844 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret podName:b9842d85-1f4d-4c40-a1ff-2254160fdb9e nodeName:}" failed. No retries permitted until 2026-07-10 17:11:32.085821693 +0000 UTC m=+18.155018778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret") pod "global-pull-secret-syncer-pw595" (UID: "b9842d85-1f4d-4c40-a1ff-2254160fdb9e") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:24.186246 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:24.186214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:24.186423 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.186380 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:24.186465 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.186439 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs podName:078c83e5-04a5-474a-8a60-4a0b8aa280d2 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:32.186423319 +0000 UTC m=+18.255620398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs") pod "network-metrics-daemon-cks5v" (UID: "078c83e5-04a5-474a-8a60-4a0b8aa280d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:24.287326 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:24.287292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:24.287519 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.287498 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:24.287600 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.287525 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:24.287600 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.287537 2580 projected.go:196] Error preparing data for projected volume kube-api-access-4smwb for pod openshift-network-diagnostics/network-check-target-95wjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:24.287735 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.287629 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb podName:9507138b-069a-4e78-8029-41513a50465d nodeName:}" failed. No retries permitted until 2026-07-10 17:11:32.287611739 +0000 UTC m=+18.356808818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4smwb" (UniqueName: "kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb") pod "network-check-target-95wjr" (UID: "9507138b-069a-4e78-8029-41513a50465d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:24.496499 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:24.496466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:24.496664 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.496590 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:24.497808 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:24.497779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:24.497927 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.497886 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:24.497991 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:24.497930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:24.498038 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:24.497996 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:26.499864 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:26.499835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:26.500309 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:26.499947 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:26.500309 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:26.499954 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:26.500309 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:26.499985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:26.500309 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:26.500094 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:26.500309 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:26.500172 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:28.496278 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:28.496240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:28.496278 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:28.496257 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:28.496814 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:28.496371 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:28.496814 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:28.496422 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:28.496814 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:28.496461 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:28.496814 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:28.496567 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:30.496535 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:30.496499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:30.496979 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:30.496510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:30.496979 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:30.496626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:30.496979 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:30.496635 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:30.496979 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:30.496735 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:30.496979 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:30.496820 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:32.148439 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:32.148395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:32.148955 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.148556 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:32.148955 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.148640 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret podName:b9842d85-1f4d-4c40-a1ff-2254160fdb9e nodeName:}" failed. No retries permitted until 2026-07-10 17:11:48.148618285 +0000 UTC m=+34.217815364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret") pod "global-pull-secret-syncer-pw595" (UID: "b9842d85-1f4d-4c40-a1ff-2254160fdb9e") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:32.249601 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:32.249572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:32.249764 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.249685 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:32.249764 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.249750 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs podName:078c83e5-04a5-474a-8a60-4a0b8aa280d2 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:48.249731932 +0000 UTC m=+34.318929028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs") pod "network-metrics-daemon-cks5v" (UID: "078c83e5-04a5-474a-8a60-4a0b8aa280d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:32.350356 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:32.350320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:32.350523 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.350501 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:32.350594 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.350527 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:32.350594 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.350538 2580 projected.go:196] Error preparing data for projected volume kube-api-access-4smwb for pod openshift-network-diagnostics/network-check-target-95wjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:32.350692 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.350603 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb podName:9507138b-069a-4e78-8029-41513a50465d nodeName:}" failed. No retries permitted until 2026-07-10 17:11:48.35057443 +0000 UTC m=+34.419771517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4smwb" (UniqueName: "kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb") pod "network-check-target-95wjr" (UID: "9507138b-069a-4e78-8029-41513a50465d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:32.496144 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:32.496068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:32.496144 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:32.496102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:32.496358 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:32.496071 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:32.496358 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.496211 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:32.496358 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.496289 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:32.496481 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:32.496365 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:34.496820 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:34.496791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:34.497259 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:34.496882 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:34.497259 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:34.496941 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:34.497259 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:34.497058 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:34.497259 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:34.497102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:34.497259 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:34.497210 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:36.498412 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.498133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:36.498909 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:36.498514 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:36.498909 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.498377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:36.498909 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:36.498672 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:36.498909 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.498717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:36.498909 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:36.498820 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:36.678324 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.678286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clvg4" event={"ID":"2f9594ea-c699-482c-914d-d8dd3803ea38","Type":"ContainerStarted","Data":"b030f0e39167af5ce80803f015939a5cb940c4fb726f0a8e429bea026b0e92bf"} Jul 10 17:11:36.679758 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.679731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4874w" event={"ID":"9a301d47-a92b-46d1-aaf9-08ecf8ffccd6","Type":"ContainerStarted","Data":"9561bea0db14d1363ea3c5dacbca3b1b58b4816981ba79a526fb1dd832f92e21"} Jul 10 17:11:36.681108 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.681086 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" event={"ID":"671bb016-c27a-48ce-8aaa-408d5f05999f","Type":"ContainerStarted","Data":"557e9f4245d3e1d1b61a934e3f2cd60c3d186fd065b7ac730677ba603c58f45b"} Jul 10 17:11:36.682412 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.682386 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n6672" event={"ID":"6e9ce868-b07d-4674-be8b-3f27b781f174","Type":"ContainerStarted","Data":"4585bf455ca85010a39eae0808bb9a7624993920a1a10cd4c6e9738459f6b61f"} Jul 10 17:11:36.683952 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.683906 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-69wk5" event={"ID":"2667319a-4b88-41ab-bba5-44f2b50e4269","Type":"ContainerStarted","Data":"c6c3503e81162dc959a39a455f81670bc9ef0842dda8d44ff06a3ea3ae75762f"} Jul 10 17:11:36.685368 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.685254 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerStarted","Data":"b67cc1c9ad21ba04ec0c59edaf1beead6886670cdfa42e39c26acf749316ce02"} Jul 10 17:11:36.687108 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.687066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m5n45" event={"ID":"4942f89d-529f-4889-82e4-0056933168ed","Type":"ContainerStarted","Data":"1b0ad4b1b406834783d6018291fc932c5a1d211886fa145a8b92df3827c26b37"} Jul 10 17:11:36.688296 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.688268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" event={"ID":"0b5dbde9-7175-4f6a-b833-2891f1286aeb","Type":"ContainerStarted","Data":"664faa5e3065bdae6328b345dd07dcaf40772dafdb4518253ab581fc4abb85a1"} Jul 10 17:11:36.731874 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.731541 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-clvg4" podStartSLOduration=4.076606689 podStartE2EDuration="22.73152508s" totalImagesPullingTime="18.654918391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:36.707085788 +0000 UTC m=+22.776282889" watchObservedRunningTime="2026-07-10 17:11:36.73152508 +0000 UTC m=+22.800722180" Jul 10 17:11:36.769081 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.769037 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n6672" podStartSLOduration=4.108676401 podStartE2EDuration="22.769021769s" totalImagesPullingTime="18.660345368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:36.747805166 +0000 UTC m=+22.817002267" watchObservedRunningTime="2026-07-10 17:11:36.769021769 +0000 UTC m=+22.838218869" Jul 10 17:11:36.769235 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.769208 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m5n45" podStartSLOduration=4.103268235 podStartE2EDuration="22.769198602s" totalImagesPullingTime="18.665930367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:36.768670181 +0000 UTC m=+22.837867281" watchObservedRunningTime="2026-07-10 17:11:36.769198602 +0000 UTC m=+22.838395703" Jul 10 17:11:36.794344 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.794305 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-69wk5" podStartSLOduration=11.522180955 podStartE2EDuration="22.794290387s" totalImagesPullingTime="11.272109432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:36.793807374 +0000 UTC m=+22.863004477" watchObservedRunningTime="2026-07-10 17:11:36.794290387 +0000 UTC m=+22.863487486" Jul 10 17:11:36.820248 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.820211 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4874w" podStartSLOduration=4.163739318 podStartE2EDuration="22.820196618s" totalImagesPullingTime="18.6564573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:36.819721171 +0000 UTC m=+22.888918272" watchObservedRunningTime="2026-07-10 17:11:36.820196618 +0000 UTC m=+22.889393718" Jul 10 17:11:36.941401 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:36.941373 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:37.030781 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.030735 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:37.130933 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.130904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:37.195030 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.194828 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:37.565763 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.565728 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:37.636294 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.636217 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:37.692469 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.692429 2580 generic.go:356] "Generic (PLEG): container finished" podID="671bb016-c27a-48ce-8aaa-408d5f05999f" containerID="557e9f4245d3e1d1b61a934e3f2cd60c3d186fd065b7ac730677ba603c58f45b" exitCode=0 Jul 10 17:11:37.692667 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.692548 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" event={"ID":"671bb016-c27a-48ce-8aaa-408d5f05999f","Type":"ContainerDied","Data":"557e9f4245d3e1d1b61a934e3f2cd60c3d186fd065b7ac730677ba603c58f45b"} Jul 10 17:11:37.695870 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.695847 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:11:37.696284 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.696258 2580 generic.go:356] "Generic (PLEG): container finished" podID="ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1" containerID="67de749dfd1b5a841d45b8520f7b680e9c4fef4d8f890a54b54840f67b114e90" exitCode=1 Jul 10 17:11:37.696409 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.696379 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerStarted","Data":"6757264ff323195bc43fa3081ca2cf1793c4f52c11cd1067798d3c679ce782a2"} Jul 10 17:11:37.696518 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.696458 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerStarted","Data":"bfbc46987549952af45537d5bb064b5e87360ae839666c0d6a8df53788809af9"} Jul 10 17:11:37.696518 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.696508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerStarted","Data":"55e652aaffdc09ffb203c92283a67ce9401dd8779f58c2ed5dfe299d7141b103"} Jul 10 17:11:37.696603 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.696554 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerStarted","Data":"3ba78f6913add97164a7361efa4990898e6d41e54321db844e6f6a756a19a91c"} Jul 10 17:11:37.696603 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.696588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerDied","Data":"67de749dfd1b5a841d45b8520f7b680e9c4fef4d8f890a54b54840f67b114e90"} Jul 10 17:11:37.808966 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:37.808847 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 17:11:38.458786 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.458675 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T17:11:37.808961455Z","UUID":"1f059333-b0a8-41e6-86de-c582213bcaa2","Handler":null,"Name":"","Endpoint":""} Jul 10 17:11:38.460461 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.460438 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 17:11:38.460584 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.460484 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 17:11:38.496114 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.495691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:38.496114 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.495691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:38.496114 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:38.495833 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:38.496114 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:38.495869 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:38.496114 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.495917 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:38.496114 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:38.495980 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:38.700193 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.700160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xgxk" event={"ID":"7e9c8570-8386-4aad-bbbb-54b00f6ad38b","Type":"ContainerStarted","Data":"805217df25b6da43d23339d58292f34c834ef09b71da93c358942c4d935dbf64"} Jul 10 17:11:38.702206 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.702181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" event={"ID":"0b5dbde9-7175-4f6a-b833-2891f1286aeb","Type":"ContainerStarted","Data":"34bb0176197e09ca16aa985ff2f0530ca3b47a4048553f02425eccf0819d61df"} Jul 10 17:11:38.716936 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.716848 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7xgxk" podStartSLOduration=6.065016366 podStartE2EDuration="24.716834348s" totalImagesPullingTime="18.651817982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:38.716405542 +0000 UTC m=+24.785602643" watchObservedRunningTime="2026-07-10 17:11:38.716834348 +0000 UTC m=+24.786031448" Jul 10 17:11:38.876131 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.876087 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:38.953805 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:38.953760 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:39.536567 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:39.536533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:39.634621 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:39.634586 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:39.708371 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:39.708347 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:11:39.708962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:39.708801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerStarted","Data":"fc25c5eecc11d387d9b1926d0d45a1740f7fc610efce0e6886b8dceb6d2b4ad9"} Jul 10 17:11:39.711226 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:39.711194 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" event={"ID":"0b5dbde9-7175-4f6a-b833-2891f1286aeb","Type":"ContainerStarted","Data":"22e2eee63daebbfa1f2aa3e9322fd218966661f6cc6489ba31ccb1ca6cd29fc6"} Jul 10 17:11:39.733291 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:39.733249 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkb2n" podStartSLOduration=4.415643809 podStartE2EDuration="25.733235317s" totalImagesPullingTime="21.317591508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:39.73179639 +0000 UTC m=+25.800993490" watchObservedRunningTime="2026-07-10 17:11:39.733235317 +0000 UTC m=+25.802432422" Jul 10 17:11:39.972238 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:39.972210 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:40.072260 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:40.072223 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:40.496097 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:40.496067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:40.496292 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:40.496068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:40.496292 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:40.496191 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:40.496292 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:40.496077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:40.496292 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:40.496267 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:40.496518 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:40.496316 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:41.160995 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:41.160961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:41.161611 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:41.161574 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:41.715761 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:41.715732 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:41.716229 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:41.716211 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4874w" Jul 10 17:11:41.787812 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:41.787785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:42.495768 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.495747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:42.496206 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.495770 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:42.496206 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:42.495855 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:42.496206 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:42.495931 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:42.496206 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.495985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:42.496206 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:42.496063 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:42.721137 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.721108 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:11:42.721566 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.721540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerStarted","Data":"82fb423d4fba920fade33dbbcb9c5fb41755685bf6cb4f52a185f12bbe60c06a"} Jul 10 17:11:42.721799 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.721782 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:42.721994 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.721976 2580 scope.go:117] "RemoveContainer" containerID="67de749dfd1b5a841d45b8520f7b680e9c4fef4d8f890a54b54840f67b114e90" Jul 10 17:11:42.723720 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.723682 2580 generic.go:356] "Generic (PLEG): container finished" podID="671bb016-c27a-48ce-8aaa-408d5f05999f" containerID="39b2ed16e3b69cffac0943a320edf6252877368730f1595066a960039f544abc" exitCode=0 Jul 10 17:11:42.723795 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.723743 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" event={"ID":"671bb016-c27a-48ce-8aaa-408d5f05999f","Type":"ContainerDied","Data":"39b2ed16e3b69cffac0943a320edf6252877368730f1595066a960039f544abc"} Jul 10 17:11:42.739721 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:42.739682 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:43.350350 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:43.350319 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:43.416094 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:43.416061 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:43.729807 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:43.729783 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:11:43.730370 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:43.730244 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" event={"ID":"ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1","Type":"ContainerStarted","Data":"fa08fdf96ae5406e90e427f7ab0fce7b604803e87cf7283e91a0ac15eb53c162"} Jul 10 17:11:43.730873 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:43.730442 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:43.730873 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:43.730521 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:43.750394 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:43.750213 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:11:43.759722 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:43.759380 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" podStartSLOduration=10.983569528 podStartE2EDuration="29.759360799s" totalImagesPullingTime="18.775791271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:43.757885486 +0000 UTC m=+29.827082586" watchObservedRunningTime="2026-07-10 17:11:43.759360799 +0000 UTC m=+29.828557899" Jul 10 17:11:44.072888 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.072856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:44.211088 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.211056 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pw595"] Jul 10 17:11:44.211219 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.211191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:44.211292 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:44.211275 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:44.214456 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.214331 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95wjr"] Jul 10 17:11:44.214456 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.214446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:44.214652 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:44.214525 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:44.214964 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.214944 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cks5v"] Jul 10 17:11:44.215042 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.215031 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:44.215133 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:44.215114 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:44.330661 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.330605 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:44.452286 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.452259 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:44.734597 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.734557 2580 generic.go:356] "Generic (PLEG): container finished" podID="671bb016-c27a-48ce-8aaa-408d5f05999f" containerID="b4685be76b436a68d671c92844e1f242b32f5f84fb01db3d46d461011947f2ad" exitCode=0 Jul 10 17:11:44.735418 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.734639 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" event={"ID":"671bb016-c27a-48ce-8aaa-408d5f05999f","Type":"ContainerDied","Data":"b4685be76b436a68d671c92844e1f242b32f5f84fb01db3d46d461011947f2ad"} Jul 10 17:11:44.778007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.777972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:44.877328 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:44.877300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:45.040345 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:45.040278 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:45.141712 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:45.141683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:45.495832 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:45.495802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:45.496000 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:45.495919 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:45.496000 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:45.495971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:45.496087 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:45.496068 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:45.765004 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:45.764913 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:45.807127 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:45.807097 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n6672_6e9ce868-b07d-4674-be8b-3f27b781f174/dns-node-resolver/0.log" Jul 10 17:11:45.864490 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:45.864455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:46.496413 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:46.496337 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:46.496550 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:46.496434 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:46.685630 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:46.685592 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:46.741306 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:46.741270 2580 generic.go:356] "Generic (PLEG): container finished" podID="671bb016-c27a-48ce-8aaa-408d5f05999f" containerID="c12066701d7c017bf9eab7913a010f847eabee8a97222ccced5e21648d9672b7" exitCode=0 Jul 10 17:11:46.741454 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:46.741318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" event={"ID":"671bb016-c27a-48ce-8aaa-408d5f05999f","Type":"ContainerDied","Data":"c12066701d7c017bf9eab7913a010f847eabee8a97222ccced5e21648d9672b7"} Jul 10 17:11:46.775563 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:46.775339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:46.805775 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:46.805756 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-69wk5_2667319a-4b88-41ab-bba5-44f2b50e4269/node-ca/0.log" Jul 10 17:11:46.873370 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:46.873343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:47.496386 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:47.496354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:47.496550 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:47.496461 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:47.496550 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:47.496515 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:47.496626 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:47.496602 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:47.748131 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:47.748007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:48.168400 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:48.168371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:48.169071 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:48.168488 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:48.169071 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:48.168550 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret podName:b9842d85-1f4d-4c40-a1ff-2254160fdb9e nodeName:}" failed. No retries permitted until 2026-07-10 17:12:20.168531526 +0000 UTC m=+66.237728619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret") pod "global-pull-secret-syncer-pw595" (UID: "b9842d85-1f4d-4c40-a1ff-2254160fdb9e") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:48.269515 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:48.269476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:48.269786 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:48.269630 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:48.269786 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:48.269699 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs podName:078c83e5-04a5-474a-8a60-4a0b8aa280d2 nodeName:}" failed. No retries permitted until 2026-07-10 17:12:20.269678949 +0000 UTC m=+66.338876041 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs") pod "network-metrics-daemon-cks5v" (UID: "078c83e5-04a5-474a-8a60-4a0b8aa280d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:48.370580 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:48.370540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:48.370766 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:48.370697 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:48.370766 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:48.370733 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:48.370766 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:48.370744 2580 projected.go:196] Error preparing data for projected volume kube-api-access-4smwb for pod openshift-network-diagnostics/network-check-target-95wjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:48.370871 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:48.370791 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb podName:9507138b-069a-4e78-8029-41513a50465d nodeName:}" failed. No retries permitted until 2026-07-10 17:12:20.37077733 +0000 UTC m=+66.439974407 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4smwb" (UniqueName: "kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb") pod "network-check-target-95wjr" (UID: "9507138b-069a-4e78-8029-41513a50465d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:48.495996 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:48.495920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:48.496128 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:48.496042 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:48.877249 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:48.877213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:48.979245 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:48.979211 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:49.495828 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:49.495792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:49.496246 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:49.495800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:49.496246 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:49.495899 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:49.496246 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:49.496016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:50.495958 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:50.495927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:50.496460 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:50.496051 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:51.496356 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:51.496288 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:51.496746 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:51.496294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:51.496746 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:51.496383 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:51.496746 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:51.496514 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:51.603580 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:51.603553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:51.743862 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:51.743830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:51.753943 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:51.753881 2580 generic.go:356] "Generic (PLEG): container finished" podID="671bb016-c27a-48ce-8aaa-408d5f05999f" containerID="d6a7d2013a0bd1bb977108b265db6b902858bbe97725f0a242d5dff8a2814b35" exitCode=0 Jul 10 17:11:51.753943 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:51.753939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" event={"ID":"671bb016-c27a-48ce-8aaa-408d5f05999f","Type":"ContainerDied","Data":"d6a7d2013a0bd1bb977108b265db6b902858bbe97725f0a242d5dff8a2814b35"} Jul 10 17:11:51.892054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:51.892029 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:52.014521 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:52.014442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:52.294890 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:52.294816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:52.416407 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:52.416343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:52.496352 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:52.496322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:52.496698 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:52.496427 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:52.524283 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:52.524253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:52.759239 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:52.759207 2580 generic.go:356] "Generic (PLEG): container finished" podID="671bb016-c27a-48ce-8aaa-408d5f05999f" containerID="7fdaa149ad695bf005a3bcfee0cf8056fad3dd16990f817438c8ed5417769ba6" exitCode=0 Jul 10 17:11:52.759390 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:52.759250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" event={"ID":"671bb016-c27a-48ce-8aaa-408d5f05999f","Type":"ContainerDied","Data":"7fdaa149ad695bf005a3bcfee0cf8056fad3dd16990f817438c8ed5417769ba6"} Jul 10 17:11:53.415543 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:53.415509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:53.495985 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:53.495947 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:53.496104 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:53.496066 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:53.496182 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:53.496132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:53.496272 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:53.496249 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:53.603567 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:53.603540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:53.764824 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:53.764753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" event={"ID":"671bb016-c27a-48ce-8aaa-408d5f05999f","Type":"ContainerStarted","Data":"b0e1742114172b7fb9682ae32b09c2040560fe6294baedcfcb211a19bdf2c5fa"} Jul 10 17:11:53.799454 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:53.799426 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:54.195826 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:54.195792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:54.496744 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:54.496653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:54.496893 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:54.496752 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:54.591982 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:54.591953 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:54.721552 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:54.721523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:54.985663 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:54.985632 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:55.185689 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:55.185631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:55.389193 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:55.389156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:55.496371 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:55.496340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:55.496497 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:55.496442 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:55.496541 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:55.496500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:55.496611 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:55.496594 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:55.587502 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:55.587469 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:55.689425 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:55.689327 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:55.819360 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:55.819285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:56.447367 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:56.447340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:56.496353 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:56.496325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:56.496462 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:56.496426 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:56.544676 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:56.544646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:56.994959 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:56.994912 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:57.185824 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:57.185793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:57.398689 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:57.398662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:57.496229 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:57.496198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:57.496385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:57.496202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:57.496385 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:57.496297 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:57.496385 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:57.496379 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:57.796671 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:57.796598 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:57.905129 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:57.905080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:58.384619 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:58.384590 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:58.495890 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:58.495865 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:11:58.496013 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:58.495970 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:11:58.525662 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:58.525637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:58.983286 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:58.983255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:59.185497 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:59.185470 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:59.287280 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:59.287203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:59.397525 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:59.397470 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:59.496400 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:59.496361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:11:59.496553 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:59.496471 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:11:59.496553 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:59.496529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:11:59.496690 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:11:59.496652 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:11:59.627132 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:11:59.627098 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:00.412845 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:00.412812 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:00.496519 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:00.496481 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:00.496694 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:00.496661 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:12:00.989139 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:00.989101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:01.188873 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:01.188841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:01.384981 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:01.384939 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:01.496331 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:01.496299 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:01.496331 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:01.496326 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:01.496767 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:01.496404 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:12:01.496767 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:01.496518 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:12:01.785695 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:01.785592 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:02.000541 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:02.000509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:02.101392 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:02.101298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:02.235370 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:02.235335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:02.311002 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:02.310966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:02.388637 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:02.388584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:02.490765 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:02.490740 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:02.496739 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:02.496721 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:02.497127 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:02.496817 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:12:02.589842 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:02.589799 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:03.496525 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:03.496498 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:03.496698 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:03.496505 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:03.496698 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:03.496661 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:12:03.496698 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:03.496582 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:12:03.514195 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:03.514165 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:03.615311 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:03.615284 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:03.692689 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:03.692667 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:03.891406 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:03.891373 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:04.024075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:04.024040 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:04.126938 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:04.126909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:04.496913 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:04.496887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:04.497303 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:04.496966 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:12:04.729766 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:04.729737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:05.496586 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:05.496557 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:05.496777 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:05.496667 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:12:05.496777 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:05.496724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:05.496858 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:05.496809 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:12:06.495759 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:06.495729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:06.496218 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:06.495815 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:12:07.496742 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:07.496686 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:07.497286 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:07.496688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:07.497286 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:07.496846 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:12:07.497286 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:07.496884 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:12:08.496410 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:08.496375 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:08.496599 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:08.496480 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:12:09.496296 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:09.496267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:09.496670 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:09.496267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:09.496670 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:09.496364 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:12:09.496670 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:09.496431 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:12:09.728975 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:09.728942 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:09.829787 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:09.829698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:10.496418 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:10.496391 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:10.496844 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:10.496480 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:12:11.496520 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:11.496469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:11.497003 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:11.496469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:11.497003 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:11.496606 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cks5v" podUID="078c83e5-04a5-474a-8a60-4a0b8aa280d2" Jul 10 17:12:11.497003 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:11.496669 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pw595" podUID="b9842d85-1f4d-4c40-a1ff-2254160fdb9e" Jul 10 17:12:11.916973 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:11.916946 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:12.021562 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:12.021531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:12.226196 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:12.226128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:12.323879 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:12.323846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:12.454431 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:12.454405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:12.496093 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:12.496039 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:12.496196 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:12.496152 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95wjr" podUID="9507138b-069a-4e78-8029-41513a50465d" Jul 10 17:12:12.603413 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:12.603388 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:12.789217 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:12.789137 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:12.889743 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:12.889698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:12.990277 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:12.990243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:13.098009 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.097920 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:13.231821 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.231797 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeReady" Jul 10 17:12:13.231972 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.231932 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 17:12:13.292323 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.292286 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9ngr2" podStartSLOduration=28.669932067 podStartE2EDuration="59.292273172s" totalImagesPullingTime="30.622341105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:53.795630237 +0000 UTC m=+39.864827337" watchObservedRunningTime="2026-07-10 17:12:13.292273172 +0000 UTC m=+59.361470272" Jul 10 17:12:13.292590 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.292576 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k57hc"] Jul 10 17:12:13.294606 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.294592 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k57hc" Jul 10 17:12:13.296538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.296518 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-52fw9"] Jul 10 17:12:13.297207 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.297191 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4zvk4\"" Jul 10 17:12:13.297724 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.297691 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 17:12:13.297833 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.297811 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 17:12:13.297896 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.297862 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 17:12:13.298250 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.298235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.300605 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.300589 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 17:12:13.300947 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.300930 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 17:12:13.301050 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.300959 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 17:12:13.301111 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.301074 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-strpp\"" Jul 10 17:12:13.301200 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.301184 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 17:12:13.305786 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.305767 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k57hc"] Jul 10 17:12:13.312139 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.312118 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-52fw9"] Jul 10 17:12:13.353570 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.353522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75f6837-81ea-42a9-81c8-8808e1e8e44d-cert\") pod \"ingress-canary-k57hc\" (UID: \"a75f6837-81ea-42a9-81c8-8808e1e8e44d\") " pod="openshift-ingress-canary/ingress-canary-k57hc" Jul 10 17:12:13.353656 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.353570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caa0dfe9-a2b5-4812-869c-ba414370830f-data-volume\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.353656 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.353592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjbm\" (UniqueName: \"kubernetes.io/projected/a75f6837-81ea-42a9-81c8-8808e1e8e44d-kube-api-access-nxjbm\") pod \"ingress-canary-k57hc\" (UID: \"a75f6837-81ea-42a9-81c8-8808e1e8e44d\") " pod="openshift-ingress-canary/ingress-canary-k57hc" Jul 10 17:12:13.353656 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.353608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hf7\" (UniqueName: \"kubernetes.io/projected/caa0dfe9-a2b5-4812-869c-ba414370830f-kube-api-access-h2hf7\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.353656 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.353626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caa0dfe9-a2b5-4812-869c-ba414370830f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.353871 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.353671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caa0dfe9-a2b5-4812-869c-ba414370830f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.353871 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.353767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caa0dfe9-a2b5-4812-869c-ba414370830f-crio-socket\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.391120 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.391098 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jg4wj"] Jul 10 17:12:13.392948 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.392935 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.398449 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.398429 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rv7kz\"" Jul 10 17:12:13.398552 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.398476 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 17:12:13.399287 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.399261 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 17:12:13.409558 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.409541 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jg4wj"] Jul 10 17:12:13.454777 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.454756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caa0dfe9-a2b5-4812-869c-ba414370830f-crio-socket\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.454881 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.454793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39c353e2-486c-4f5d-a367-7a77b46fd9ba-config-volume\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.454881 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.454822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75f6837-81ea-42a9-81c8-8808e1e8e44d-cert\") pod \"ingress-canary-k57hc\" (UID: \"a75f6837-81ea-42a9-81c8-8808e1e8e44d\") " pod="openshift-ingress-canary/ingress-canary-k57hc" Jul 10 17:12:13.454881 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.454872 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caa0dfe9-a2b5-4812-869c-ba414370830f-crio-socket\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.455036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.454890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c353e2-486c-4f5d-a367-7a77b46fd9ba-metrics-tls\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.455036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.454932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5srd\" (UniqueName: \"kubernetes.io/projected/39c353e2-486c-4f5d-a367-7a77b46fd9ba-kube-api-access-l5srd\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.455036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.454959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caa0dfe9-a2b5-4812-869c-ba414370830f-data-volume\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.455036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.454982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c353e2-486c-4f5d-a367-7a77b46fd9ba-tmp-dir\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.455036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.455007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjbm\" (UniqueName: \"kubernetes.io/projected/a75f6837-81ea-42a9-81c8-8808e1e8e44d-kube-api-access-nxjbm\") pod \"ingress-canary-k57hc\" (UID: \"a75f6837-81ea-42a9-81c8-8808e1e8e44d\") " pod="openshift-ingress-canary/ingress-canary-k57hc" Jul 10 17:12:13.455036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.455032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hf7\" (UniqueName: \"kubernetes.io/projected/caa0dfe9-a2b5-4812-869c-ba414370830f-kube-api-access-h2hf7\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.455272 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.455105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caa0dfe9-a2b5-4812-869c-ba414370830f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.455272 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.455131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caa0dfe9-a2b5-4812-869c-ba414370830f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.455356 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.455292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caa0dfe9-a2b5-4812-869c-ba414370830f-data-volume\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.455614 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.455596 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caa0dfe9-a2b5-4812-869c-ba414370830f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.458923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.458903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caa0dfe9-a2b5-4812-869c-ba414370830f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.458996 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.458988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75f6837-81ea-42a9-81c8-8808e1e8e44d-cert\") pod \"ingress-canary-k57hc\" (UID: \"a75f6837-81ea-42a9-81c8-8808e1e8e44d\") " pod="openshift-ingress-canary/ingress-canary-k57hc" Jul 10 17:12:13.463953 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.463934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hf7\" (UniqueName: \"kubernetes.io/projected/caa0dfe9-a2b5-4812-869c-ba414370830f-kube-api-access-h2hf7\") pod \"insights-runtime-extractor-52fw9\" (UID: \"caa0dfe9-a2b5-4812-869c-ba414370830f\") " pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.464050 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.463958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjbm\" (UniqueName: \"kubernetes.io/projected/a75f6837-81ea-42a9-81c8-8808e1e8e44d-kube-api-access-nxjbm\") pod \"ingress-canary-k57hc\" (UID: \"a75f6837-81ea-42a9-81c8-8808e1e8e44d\") " pod="openshift-ingress-canary/ingress-canary-k57hc" Jul 10 17:12:13.496040 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.496020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:13.496136 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.496024 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:13.498885 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.498865 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 17:12:13.499037 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.499020 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rwncl\"" Jul 10 17:12:13.499110 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.499065 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 17:12:13.532754 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.532728 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:13.555792 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.555771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c353e2-486c-4f5d-a367-7a77b46fd9ba-tmp-dir\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.555865 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.555832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39c353e2-486c-4f5d-a367-7a77b46fd9ba-config-volume\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.555992 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.555975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c353e2-486c-4f5d-a367-7a77b46fd9ba-metrics-tls\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.556053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.556012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5srd\" (UniqueName: \"kubernetes.io/projected/39c353e2-486c-4f5d-a367-7a77b46fd9ba-kube-api-access-l5srd\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.556115 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.556096 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c353e2-486c-4f5d-a367-7a77b46fd9ba-tmp-dir\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.556290 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.556275 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39c353e2-486c-4f5d-a367-7a77b46fd9ba-config-volume\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.558027 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.558009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c353e2-486c-4f5d-a367-7a77b46fd9ba-metrics-tls\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.564622 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.564604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5srd\" (UniqueName: \"kubernetes.io/projected/39c353e2-486c-4f5d-a367-7a77b46fd9ba-kube-api-access-l5srd\") pod \"dns-default-jg4wj\" (UID: \"39c353e2-486c-4f5d-a367-7a77b46fd9ba\") " pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.606584 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.606528 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k57hc" Jul 10 17:12:13.611279 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.611259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-52fw9" Jul 10 17:12:13.633787 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.633689 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:13.702940 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.702917 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:13.747851 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.747820 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k57hc"] Jul 10 17:12:13.751103 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:13.751068 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75f6837_81ea_42a9_81c8_8808e1e8e44d.slice/crio-b16288a947ab7a19d8785710b8ef2b71649bc219a3add7bc7cd4e31e27c220a6 WatchSource:0}: Error finding container b16288a947ab7a19d8785710b8ef2b71649bc219a3add7bc7cd4e31e27c220a6: Status 404 returned error can't find the container with id b16288a947ab7a19d8785710b8ef2b71649bc219a3add7bc7cd4e31e27c220a6 Jul 10 17:12:13.766175 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.766125 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-52fw9"] Jul 10 17:12:13.770002 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:13.769975 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa0dfe9_a2b5_4812_869c_ba414370830f.slice/crio-dfb1841538c1523d856a170bd8075e94e74b43814fb74e4132943017db85f0da WatchSource:0}: Error finding container dfb1841538c1523d856a170bd8075e94e74b43814fb74e4132943017db85f0da: Status 404 returned error can't find the container with id dfb1841538c1523d856a170bd8075e94e74b43814fb74e4132943017db85f0da Jul 10 17:12:13.810665 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.810636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52fw9" event={"ID":"caa0dfe9-a2b5-4812-869c-ba414370830f","Type":"ContainerStarted","Data":"dfb1841538c1523d856a170bd8075e94e74b43814fb74e4132943017db85f0da"} Jul 10 17:12:13.811692 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.811669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k57hc" event={"ID":"a75f6837-81ea-42a9-81c8-8808e1e8e44d","Type":"ContainerStarted","Data":"b16288a947ab7a19d8785710b8ef2b71649bc219a3add7bc7cd4e31e27c220a6"} Jul 10 17:12:13.841524 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:13.841502 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jg4wj"] Jul 10 17:12:13.844521 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:13.844497 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c353e2_486c_4f5d_a367_7a77b46fd9ba.slice/crio-5aa6dba48b8ed3bdde147042578cf09831b79c9a4e79b144579d50cafbb7d693 WatchSource:0}: Error finding container 5aa6dba48b8ed3bdde147042578cf09831b79c9a4e79b144579d50cafbb7d693: Status 404 returned error can't find the container with id 5aa6dba48b8ed3bdde147042578cf09831b79c9a4e79b144579d50cafbb7d693 Jul 10 17:12:14.500275 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:14.500240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:14.503188 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:14.503160 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 17:12:14.504210 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:14.504018 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r2699\"" Jul 10 17:12:14.504210 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:14.504067 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 17:12:14.817403 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:14.817309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg4wj" event={"ID":"39c353e2-486c-4f5d-a367-7a77b46fd9ba","Type":"ContainerStarted","Data":"5aa6dba48b8ed3bdde147042578cf09831b79c9a4e79b144579d50cafbb7d693"} Jul 10 17:12:15.751758 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:15.751731 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jc2mr" Jul 10 17:12:15.877382 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:15.877270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:15.978304 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:15.978259 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:19.539505 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:19.539478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:19.832100 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:19.832015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k57hc" event={"ID":"a75f6837-81ea-42a9-81c8-8808e1e8e44d","Type":"ContainerStarted","Data":"0b3ba4ff10d29a43c0b159da2600bb8be3cd07a1d29e1faaaae9a83e278b9d44"} Jul 10 17:12:19.833848 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:19.833822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52fw9" event={"ID":"caa0dfe9-a2b5-4812-869c-ba414370830f","Type":"ContainerStarted","Data":"becbc139dbe2d8a10dc1057ef9d6e2dc6c6bf72ac57b23fd3520d447bf2f8d45"} Jul 10 17:12:19.833975 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:19.833874 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52fw9" event={"ID":"caa0dfe9-a2b5-4812-869c-ba414370830f","Type":"ContainerStarted","Data":"f63a3aafabe1ba7a73bf8b1cbc1f46dfff5c1be6940d95ea159fd8e1008415ee"} Jul 10 17:12:19.835576 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:19.835553 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg4wj" event={"ID":"39c353e2-486c-4f5d-a367-7a77b46fd9ba","Type":"ContainerStarted","Data":"e4ccf70fb9b6eee6184c7251f759cfef43638701d6bc9539aa362463e90e0b46"} Jul 10 17:12:19.835662 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:19.835617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg4wj" event={"ID":"39c353e2-486c-4f5d-a367-7a77b46fd9ba","Type":"ContainerStarted","Data":"bd37d225930daefab9f1952ba7f12c34dbbcb63cdb619af4c8a0edef3069d8c3"} Jul 10 17:12:19.835724 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:19.835692 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:19.849330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:19.849220 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k57hc" podStartSLOduration=1.624687059 podStartE2EDuration="6.849206882s" totalImagesPullingTime="5.224519823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:19.849058964 +0000 UTC m=+65.918256064" watchObservedRunningTime="2026-07-10 17:12:19.849206882 +0000 UTC m=+65.918403981" Jul 10 17:12:19.868198 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:19.868157 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jg4wj" podStartSLOduration=1.758174684 podStartE2EDuration="6.868146349s" totalImagesPullingTime="5.109971665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:19.867317743 +0000 UTC m=+65.936514842" watchObservedRunningTime="2026-07-10 17:12:19.868146349 +0000 UTC m=+65.937343449" Jul 10 17:12:20.206409 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.206372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:20.209608 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.209587 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 17:12:20.219665 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.219634 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9842d85-1f4d-4c40-a1ff-2254160fdb9e-original-pull-secret\") pod \"global-pull-secret-syncer-pw595\" (UID: \"b9842d85-1f4d-4c40-a1ff-2254160fdb9e\") " pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:20.306711 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.306671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:20.309632 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.309604 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 17:12:20.319433 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.319411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078c83e5-04a5-474a-8a60-4a0b8aa280d2-metrics-certs\") pod \"network-metrics-daemon-cks5v\" (UID: \"078c83e5-04a5-474a-8a60-4a0b8aa280d2\") " pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:20.394375 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.394346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:20.407735 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.407694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:20.407870 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.407852 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rwncl\"" Jul 10 17:12:20.409741 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.409719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pw595" Jul 10 17:12:20.410224 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.410204 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smwb\" (UniqueName: \"kubernetes.io/projected/9507138b-069a-4e78-8029-41513a50465d-kube-api-access-4smwb\") pod \"network-check-target-95wjr\" (UID: \"9507138b-069a-4e78-8029-41513a50465d\") " pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:20.415693 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.415675 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cks5v" Jul 10 17:12:20.435532 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.435294 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-8ch2h"] Jul 10 17:12:20.438170 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.438141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.441597 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.441578 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 17:12:20.441697 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.441631 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 17:12:20.442199 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.442030 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 10 17:12:20.442810 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.442562 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 17:12:20.442810 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.442609 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 10 17:12:20.443925 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.443763 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nhzbf\"" Jul 10 17:12:20.452306 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.452226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-8ch2h"] Jul 10 17:12:20.476191 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.476159 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:20.508422 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.508396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/860399f4-6cc7-44bb-95bd-60e1b92e5d81-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.508537 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.508444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/860399f4-6cc7-44bb-95bd-60e1b92e5d81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.508537 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.508478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kw82\" (UniqueName: \"kubernetes.io/projected/860399f4-6cc7-44bb-95bd-60e1b92e5d81-kube-api-access-8kw82\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.508537 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.508527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860399f4-6cc7-44bb-95bd-60e1b92e5d81-metrics-client-ca\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.513757 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.513739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:20.560388 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.560352 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pw595"] Jul 10 17:12:20.565104 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:20.565064 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9842d85_1f4d_4c40_a1ff_2254160fdb9e.slice/crio-dbf7bfab253be2a138e2b66213798c64efa810b3fa7de206d3ff5e274964c10f WatchSource:0}: Error finding container dbf7bfab253be2a138e2b66213798c64efa810b3fa7de206d3ff5e274964c10f: Status 404 returned error can't find the container with id dbf7bfab253be2a138e2b66213798c64efa810b3fa7de206d3ff5e274964c10f Jul 10 17:12:20.577742 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.577694 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cks5v"] Jul 10 17:12:20.610800 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.610290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/860399f4-6cc7-44bb-95bd-60e1b92e5d81-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.610800 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:20.610690 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 10 17:12:20.610800 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:20.610782 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/860399f4-6cc7-44bb-95bd-60e1b92e5d81-prometheus-operator-tls podName:860399f4-6cc7-44bb-95bd-60e1b92e5d81 nodeName:}" failed. No retries permitted until 2026-07-10 17:12:21.110759482 +0000 UTC m=+67.179956580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/860399f4-6cc7-44bb-95bd-60e1b92e5d81-prometheus-operator-tls") pod "prometheus-operator-64785b6675-8ch2h" (UID: "860399f4-6cc7-44bb-95bd-60e1b92e5d81") : secret "prometheus-operator-tls" not found Jul 10 17:12:20.612144 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.611165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/860399f4-6cc7-44bb-95bd-60e1b92e5d81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.612144 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.611235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kw82\" (UniqueName: \"kubernetes.io/projected/860399f4-6cc7-44bb-95bd-60e1b92e5d81-kube-api-access-8kw82\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.612144 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.611300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860399f4-6cc7-44bb-95bd-60e1b92e5d81-metrics-client-ca\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.612144 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.612077 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860399f4-6cc7-44bb-95bd-60e1b92e5d81-metrics-client-ca\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.613867 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.613843 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/860399f4-6cc7-44bb-95bd-60e1b92e5d81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.630917 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.630891 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kw82\" (UniqueName: \"kubernetes.io/projected/860399f4-6cc7-44bb-95bd-60e1b92e5d81-kube-api-access-8kw82\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:20.666010 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.665978 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95wjr"] Jul 10 17:12:20.669511 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:20.669480 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9507138b_069a_4e78_8029_41513a50465d.slice/crio-c7c1c2038bfe2f89fd9548d80c8a2e9a9c5767d7af862dacb78f6ee7f7d69e0a WatchSource:0}: Error finding container c7c1c2038bfe2f89fd9548d80c8a2e9a9c5767d7af862dacb78f6ee7f7d69e0a: Status 404 returned error can't find the container with id c7c1c2038bfe2f89fd9548d80c8a2e9a9c5767d7af862dacb78f6ee7f7d69e0a Jul 10 17:12:20.709352 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.709320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:20.841425 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.841343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95wjr" event={"ID":"9507138b-069a-4e78-8029-41513a50465d","Type":"ContainerStarted","Data":"c7c1c2038bfe2f89fd9548d80c8a2e9a9c5767d7af862dacb78f6ee7f7d69e0a"} Jul 10 17:12:20.842684 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.842659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cks5v" event={"ID":"078c83e5-04a5-474a-8a60-4a0b8aa280d2","Type":"ContainerStarted","Data":"a45380f55b53a9a94216299b0a491daa7ceefb459d289f413bd4bb7794e96293"} Jul 10 17:12:20.843881 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:20.843850 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pw595" event={"ID":"b9842d85-1f4d-4c40-a1ff-2254160fdb9e","Type":"ContainerStarted","Data":"dbf7bfab253be2a138e2b66213798c64efa810b3fa7de206d3ff5e274964c10f"} Jul 10 17:12:21.115490 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:21.115404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/860399f4-6cc7-44bb-95bd-60e1b92e5d81-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:21.117943 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:21.117921 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/860399f4-6cc7-44bb-95bd-60e1b92e5d81-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8ch2h\" (UID: \"860399f4-6cc7-44bb-95bd-60e1b92e5d81\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:21.351305 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:21.351263 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" Jul 10 17:12:21.829838 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:21.829801 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:21.908064 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:21.908020 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:22.438131 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:22.438016 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-8ch2h"] Jul 10 17:12:22.441251 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:22.441111 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860399f4_6cc7_44bb_95bd_60e1b92e5d81.slice/crio-a4a004d9269804a17aa86bfe8d9955b8ee3a6674147297804dd25b8c948e901b WatchSource:0}: Error finding container a4a004d9269804a17aa86bfe8d9955b8ee3a6674147297804dd25b8c948e901b: Status 404 returned error can't find the container with id a4a004d9269804a17aa86bfe8d9955b8ee3a6674147297804dd25b8c948e901b Jul 10 17:12:22.853558 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:22.853463 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cks5v" event={"ID":"078c83e5-04a5-474a-8a60-4a0b8aa280d2","Type":"ContainerStarted","Data":"fd60665121085c7419c19bb6cd70cc2c0ed28c7e902dc839195f99e5566b81d6"} Jul 10 17:12:22.853558 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:22.853523 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cks5v" event={"ID":"078c83e5-04a5-474a-8a60-4a0b8aa280d2","Type":"ContainerStarted","Data":"3eefffdc43c5ba15bd90d2b2e72bb611ee77110d3efb19f591512a500cb91650"} Jul 10 17:12:22.855059 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:22.855028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" event={"ID":"860399f4-6cc7-44bb-95bd-60e1b92e5d81","Type":"ContainerStarted","Data":"a4a004d9269804a17aa86bfe8d9955b8ee3a6674147297804dd25b8c948e901b"} Jul 10 17:12:22.856841 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:22.856814 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52fw9" event={"ID":"caa0dfe9-a2b5-4812-869c-ba414370830f","Type":"ContainerStarted","Data":"364db1ed80581459807e9b527514df3fc69af3e8a00b13dc1df469d12737f71f"} Jul 10 17:12:22.871572 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:22.871499 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cks5v" podStartSLOduration=67.162591056 podStartE2EDuration="1m8.8714809s" totalImagesPullingTime="1.708889844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:22.870271313 +0000 UTC m=+68.939468417" watchObservedRunningTime="2026-07-10 17:12:22.8714809 +0000 UTC m=+68.940678001" Jul 10 17:12:22.892180 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:22.892138 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-52fw9" podStartSLOduration=1.507019681 podStartE2EDuration="9.892124103s" totalImagesPullingTime="8.385104422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:12:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:22.891542694 +0000 UTC m=+68.960739794" watchObservedRunningTime="2026-07-10 17:12:22.892124103 +0000 UTC m=+68.961321203" Jul 10 17:12:22.933654 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:22.933616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:22.999130 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:22.999106 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:23.100663 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:23.100631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:26.134087 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.134049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:26.210992 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.210956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:26.647527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.647489 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:26.749168 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.749129 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:26.848431 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.848353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:26.873002 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.872975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95wjr" event={"ID":"9507138b-069a-4e78-8029-41513a50465d","Type":"ContainerStarted","Data":"a8ebab570c6d7a2ee09645222bbafd9d7612cb4dff05e833c6ff7d818fdb5b45"} Jul 10 17:12:26.873114 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.873081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:26.874367 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.874343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pw595" event={"ID":"b9842d85-1f4d-4c40-a1ff-2254160fdb9e","Type":"ContainerStarted","Data":"e5ac2a527482dcb91d2e7b4ea8774f93ba3a153b247a2b7416d93039c9abb986"} Jul 10 17:12:26.876025 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.876006 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" event={"ID":"860399f4-6cc7-44bb-95bd-60e1b92e5d81","Type":"ContainerStarted","Data":"157ec28b4a97f04af0bb0aa989f24b1da0be8ccdf21f05c9081fb566ab4b0cee"} Jul 10 17:12:26.876131 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.876042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" event={"ID":"860399f4-6cc7-44bb-95bd-60e1b92e5d81","Type":"ContainerStarted","Data":"26f9c87598ac8d7a218d2f3b7efaccbe17364b2442e36f70ca32e83c6c0ba8e9"} Jul 10 17:12:26.891580 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.891542 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-95wjr" podStartSLOduration=67.023727464 podStartE2EDuration="1m12.891531381s" totalImagesPullingTime="5.867803917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:26.89022866 +0000 UTC m=+72.959425759" watchObservedRunningTime="2026-07-10 17:12:26.891531381 +0000 UTC m=+72.960728481" Jul 10 17:12:26.911451 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.911406 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-8ch2h" podStartSLOduration=2.82844921 podStartE2EDuration="6.911390016s" totalImagesPullingTime="4.082940806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:26.910565714 +0000 UTC m=+72.979762815" watchObservedRunningTime="2026-07-10 17:12:26.911390016 +0000 UTC m=+72.980587118" Jul 10 17:12:26.926554 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:26.926515 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pw595" podStartSLOduration=66.952826079 podStartE2EDuration="1m12.926505745s" totalImagesPullingTime="5.973679666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:26.925731074 +0000 UTC m=+72.994928171" watchObservedRunningTime="2026-07-10 17:12:26.926505745 +0000 UTC m=+72.995702871" Jul 10 17:12:28.558011 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.557978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:28.804651 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.804619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:28.835803 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.835694 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt"] Jul 10 17:12:28.838190 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.838175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:28.841180 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.841164 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pv5g6\"" Jul 10 17:12:28.842092 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.842078 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 17:12:28.842184 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.842099 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 17:12:28.856534 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.856517 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt"] Jul 10 17:12:28.866560 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.866542 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-42jvt"] Jul 10 17:12:28.868585 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.868570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:28.873220 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.873205 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rzttg\"" Jul 10 17:12:28.874866 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.874848 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 17:12:28.875018 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.875002 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 17:12:28.890017 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.890001 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 17:12:28.892691 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.892671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-42jvt"] Jul 10 17:12:28.946049 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.946022 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nhfq9"] Jul 10 17:12:28.948471 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.948440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:28.950939 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.950913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:28.954242 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.954224 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ngsl6\"" Jul 10 17:12:28.954555 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.954540 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 17:12:28.954787 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.954774 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 17:12:28.955081 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.955066 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 17:12:28.971746 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7cx\" (UniqueName: \"kubernetes.io/projected/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-api-access-rl7cx\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:28.971841 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:28.971841 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:28.971841 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x479d\" (UniqueName: \"kubernetes.io/projected/accfc9e9-b413-4f75-bcbe-eae0da6c8370-kube-api-access-x479d\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:28.971960 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/accfc9e9-b413-4f75-bcbe-eae0da6c8370-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:28.971960 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/accfc9e9-b413-4f75-bcbe-eae0da6c8370-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:28.971960 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:28.971960 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971906 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:28.971960 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:28.972110 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:28.971981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/accfc9e9-b413-4f75-bcbe-eae0da6c8370-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.035439 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.035409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:29.072822 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.072800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7cx\" (UniqueName: \"kubernetes.io/projected/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-api-access-rl7cx\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.072915 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.072828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/accfc9e9-b413-4f75-bcbe-eae0da6c8370-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.072915 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.072844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.072915 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.072864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvht6\" (UniqueName: \"kubernetes.io/projected/f68bc723-68a6-4a35-9cf9-4c79895b02cd-kube-api-access-nvht6\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.072915 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.072887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.072915 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.072913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f68bc723-68a6-4a35-9cf9-4c79895b02cd-metrics-client-ca\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.073171 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.073171 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x479d\" (UniqueName: \"kubernetes.io/projected/accfc9e9-b413-4f75-bcbe-eae0da6c8370-kube-api-access-x479d\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.073171 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.073171 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/accfc9e9-b413-4f75-bcbe-eae0da6c8370-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.073171 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f68bc723-68a6-4a35-9cf9-4c79895b02cd-sys\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.073411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/accfc9e9-b413-4f75-bcbe-eae0da6c8370-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.073411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.073411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f68bc723-68a6-4a35-9cf9-4c79895b02cd-root\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.073411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.073411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.073411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-wtmp\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.073411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-textfile\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.073783 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-tls\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.073783 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.073783 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/accfc9e9-b413-4f75-bcbe-eae0da6c8370-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.073783 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.073753 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.074170 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.074149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.075523 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.075498 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.075599 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.075544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.075947 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.075924 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/accfc9e9-b413-4f75-bcbe-eae0da6c8370-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.076108 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.076090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/accfc9e9-b413-4f75-bcbe-eae0da6c8370-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.081290 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.081260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x479d\" (UniqueName: \"kubernetes.io/projected/accfc9e9-b413-4f75-bcbe-eae0da6c8370-kube-api-access-x479d\") pod \"openshift-state-metrics-84bd68b896-tdsvt\" (UID: \"accfc9e9-b413-4f75-bcbe-eae0da6c8370\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.081364 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.081293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7cx\" (UniqueName: \"kubernetes.io/projected/c971946f-e6b7-4b3e-9d1c-54109d5dbd86-kube-api-access-rl7cx\") pod \"kube-state-metrics-785778ff74-42jvt\" (UID: \"c971946f-e6b7-4b3e-9d1c-54109d5dbd86\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.146819 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.146799 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" Jul 10 17:12:29.168331 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.168300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:29.174282 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvht6\" (UniqueName: \"kubernetes.io/projected/f68bc723-68a6-4a35-9cf9-4c79895b02cd-kube-api-access-nvht6\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174372 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f68bc723-68a6-4a35-9cf9-4c79895b02cd-metrics-client-ca\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174372 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f68bc723-68a6-4a35-9cf9-4c79895b02cd-sys\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174445 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f68bc723-68a6-4a35-9cf9-4c79895b02cd-root\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174564 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-wtmp\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174564 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-textfile\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174564 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-tls\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174879 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f68bc723-68a6-4a35-9cf9-4c79895b02cd-root\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174969 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-wtmp\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.174969 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.174918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f68bc723-68a6-4a35-9cf9-4c79895b02cd-sys\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.175053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.175015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f68bc723-68a6-4a35-9cf9-4c79895b02cd-metrics-client-ca\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.175053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.175023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-textfile\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.175147 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.175117 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.176046 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.176028 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" Jul 10 17:12:29.176737 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.176689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.176896 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.176878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f68bc723-68a6-4a35-9cf9-4c79895b02cd-node-exporter-tls\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.185360 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.185317 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvht6\" (UniqueName: \"kubernetes.io/projected/f68bc723-68a6-4a35-9cf9-4c79895b02cd-kube-api-access-nvht6\") pod \"node-exporter-nhfq9\" (UID: \"f68bc723-68a6-4a35-9cf9-4c79895b02cd\") " pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.260211 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.260172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nhfq9" Jul 10 17:12:29.269172 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.269144 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:29.272261 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:29.272236 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68bc723_68a6_4a35_9cf9_4c79895b02cd.slice/crio-ded37f53321665f918c4722c1183af1428a981567487c7e1f2eb75b585768dbf WatchSource:0}: Error finding container ded37f53321665f918c4722c1183af1428a981567487c7e1f2eb75b585768dbf: Status 404 returned error can't find the container with id ded37f53321665f918c4722c1183af1428a981567487c7e1f2eb75b585768dbf Jul 10 17:12:29.286891 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.286867 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt"] Jul 10 17:12:29.290023 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:29.289989 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccfc9e9_b413_4f75_bcbe_eae0da6c8370.slice/crio-532bd11870fe8cc2aeb2412d9c899effe77328d620957532b7faab07a4002109 WatchSource:0}: Error finding container 532bd11870fe8cc2aeb2412d9c899effe77328d620957532b7faab07a4002109: Status 404 returned error can't find the container with id 532bd11870fe8cc2aeb2412d9c899effe77328d620957532b7faab07a4002109 Jul 10 17:12:29.312091 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.312072 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-42jvt"] Jul 10 17:12:29.325575 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:29.325552 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc971946f_e6b7_4b3e_9d1c_54109d5dbd86.slice/crio-7b2a0503c16983bbd97bbcee00b72d8ecebc1255f8f6b266856acd39975dcdaa WatchSource:0}: Error finding container 7b2a0503c16983bbd97bbcee00b72d8ecebc1255f8f6b266856acd39975dcdaa: Status 404 returned error can't find the container with id 7b2a0503c16983bbd97bbcee00b72d8ecebc1255f8f6b266856acd39975dcdaa Jul 10 17:12:29.372360 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.372339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:29.814324 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.814233 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:29.846201 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.846176 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jg4wj" Jul 10 17:12:29.886762 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.886733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" event={"ID":"c971946f-e6b7-4b3e-9d1c-54109d5dbd86","Type":"ContainerStarted","Data":"7b2a0503c16983bbd97bbcee00b72d8ecebc1255f8f6b266856acd39975dcdaa"} Jul 10 17:12:29.888095 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.888067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nhfq9" event={"ID":"f68bc723-68a6-4a35-9cf9-4c79895b02cd","Type":"ContainerStarted","Data":"ded37f53321665f918c4722c1183af1428a981567487c7e1f2eb75b585768dbf"} Jul 10 17:12:29.890205 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.890181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" event={"ID":"accfc9e9-b413-4f75-bcbe-eae0da6c8370","Type":"ContainerStarted","Data":"df6bec38f8a4a42ab59be42df541ded0fa1a6eac3aa6e6f9ba8a3edb530593b0"} Jul 10 17:12:29.890300 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.890233 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" event={"ID":"accfc9e9-b413-4f75-bcbe-eae0da6c8370","Type":"ContainerStarted","Data":"3eb6aab5d04f4040bc7ac197bd0033380c09b49a5959256c48f73d0a3362e340"} Jul 10 17:12:29.890300 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.890257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" event={"ID":"accfc9e9-b413-4f75-bcbe-eae0da6c8370","Type":"ContainerStarted","Data":"532bd11870fe8cc2aeb2412d9c899effe77328d620957532b7faab07a4002109"} Jul 10 17:12:29.912047 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.912019 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:12:29.912996 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.912972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:29.914616 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.914599 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.920484 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.920462 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 17:12:29.920597 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.920497 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 17:12:29.920597 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.920512 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 17:12:29.920726 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.920623 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 17:12:29.921424 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.921404 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 17:12:29.921497 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.921430 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 17:12:29.921497 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.921445 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 17:12:29.921608 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.921503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 17:12:29.921608 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.921447 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 17:12:29.921608 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.921507 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m8mhm\"" Jul 10 17:12:29.937235 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.937207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:12:29.982181 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982331 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982391 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982391 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-config-out\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982607 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982607 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-web-config\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982731 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982731 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982731 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982731 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qwx\" (UniqueName: \"kubernetes.io/projected/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-kube-api-access-s7qwx\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:29.982886 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:29.982757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.038014 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.037951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:30.084012 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.083955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084012 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.083983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084012 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084300 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qwx\" (UniqueName: \"kubernetes.io/projected/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-kube-api-access-s7qwx\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084300 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:30.084131 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-alertmanager-trusted-ca-bundle podName:f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3 nodeName:}" failed. No retries permitted until 2026-07-10 17:12:30.584106831 +0000 UTC m=+76.653303910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3") : configmap references non-existent config key: ca-bundle.crt Jul 10 17:12:30.084300 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084300 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084300 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084300 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084619 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-config-out\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084619 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084619 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084619 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.084619 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:30.084487 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 10 17:12:30.084619 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:30.084567 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-main-tls podName:f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3 nodeName:}" failed. No retries permitted until 2026-07-10 17:12:30.584543251 +0000 UTC m=+76.653740337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3") : secret "alertmanager-main-tls" not found Jul 10 17:12:30.084619 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.084596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-web-config\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.085821 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.085043 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.085821 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.085303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.087208 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.087165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.088884 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.088823 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.089231 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.089202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.089784 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.089742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.090262 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.089825 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-web-config\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.091040 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.090512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-config-out\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.091040 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.090632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.091040 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.090945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.093797 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.093771 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qwx\" (UniqueName: \"kubernetes.io/projected/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-kube-api-access-s7qwx\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.255083 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.255042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:30.590209 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.590178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.590342 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.590242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.591297 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.591270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.593091 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.593047 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.827021 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.826993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:12:30.895424 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:30.895385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nhfq9" event={"ID":"f68bc723-68a6-4a35-9cf9-4c79895b02cd","Type":"ContainerStarted","Data":"15fc982768ac2a3f0bafc56600169b41403ca4c0b8cdba6fedb04e2c69c741aa"} Jul 10 17:12:31.107588 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.107557 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:12:31.111395 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:31.111363 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86261f9_6ae2_4ac6_a8e3_7473d42c7cf3.slice/crio-c326441b5b2405a1592da4bd7e96658bce9159f95015958208ebc6132c58a944 WatchSource:0}: Error finding container c326441b5b2405a1592da4bd7e96658bce9159f95015958208ebc6132c58a944: Status 404 returned error can't find the container with id c326441b5b2405a1592da4bd7e96658bce9159f95015958208ebc6132c58a944 Jul 10 17:12:31.776036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.776005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:31.877661 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.877623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:31.901404 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.901367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" event={"ID":"c971946f-e6b7-4b3e-9d1c-54109d5dbd86","Type":"ContainerStarted","Data":"201c4ed3b64c25cc3f291ea0f5c70fc03f06141d4dbdbc83ac06fd4daf70148b"} Jul 10 17:12:31.901554 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.901439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" event={"ID":"c971946f-e6b7-4b3e-9d1c-54109d5dbd86","Type":"ContainerStarted","Data":"5c4aa8528a4c6cedcbec361bdbc036c67b55fd9510e0df24a908fbca2c34e43b"} Jul 10 17:12:31.901554 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.901469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" event={"ID":"c971946f-e6b7-4b3e-9d1c-54109d5dbd86","Type":"ContainerStarted","Data":"79b15e3a29a4ad6a05438f4121ca00ab2bfc7f5bc712c08dc7b968240f86b610"} Jul 10 17:12:31.903764 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.903733 2580 generic.go:356] "Generic (PLEG): container finished" podID="f68bc723-68a6-4a35-9cf9-4c79895b02cd" containerID="15fc982768ac2a3f0bafc56600169b41403ca4c0b8cdba6fedb04e2c69c741aa" exitCode=0 Jul 10 17:12:31.903928 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.903844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nhfq9" event={"ID":"f68bc723-68a6-4a35-9cf9-4c79895b02cd","Type":"ContainerDied","Data":"15fc982768ac2a3f0bafc56600169b41403ca4c0b8cdba6fedb04e2c69c741aa"} Jul 10 17:12:31.906780 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.906691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" event={"ID":"accfc9e9-b413-4f75-bcbe-eae0da6c8370","Type":"ContainerStarted","Data":"52fd608f93ce56eed214a35b5cbfdf0a63e9549e282abd49136a64d1d900d7b9"} Jul 10 17:12:31.908685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.908658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3","Type":"ContainerStarted","Data":"c326441b5b2405a1592da4bd7e96658bce9159f95015958208ebc6132c58a944"} Jul 10 17:12:31.916614 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.916591 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f899fbcd8-t77w7"] Jul 10 17:12:31.920187 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.920167 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:31.923289 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.923049 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7vql7jv9kbgpq\"" Jul 10 17:12:31.923289 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.923064 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 17:12:31.923289 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.923113 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 17:12:31.923289 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.923132 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 17:12:31.923289 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.923053 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 17:12:31.923596 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.923411 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 17:12:31.923596 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.923515 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-f2pfp\"" Jul 10 17:12:31.927977 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.927930 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-42jvt" podStartSLOduration=2.2711043699999998 podStartE2EDuration="3.927913187s" totalImagesPullingTime="1.656808817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:31.926885988 +0000 UTC m=+77.996083103" watchObservedRunningTime="2026-07-10 17:12:31.927913187 +0000 UTC m=+77.997110288" Jul 10 17:12:31.933600 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.933527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f899fbcd8-t77w7"] Jul 10 17:12:31.954523 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:31.954482 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tdsvt" podStartSLOduration=2.754454993 podStartE2EDuration="3.954464858s" totalImagesPullingTime="1.200009865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:31.953795299 +0000 UTC m=+78.022992403" watchObservedRunningTime="2026-07-10 17:12:31.954464858 +0000 UTC m=+78.023661959" Jul 10 17:12:32.001923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.000516 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.001923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.000575 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.001923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.000595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.001923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.000612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvrx\" (UniqueName: \"kubernetes.io/projected/4caec5d7-6528-4315-9b54-469e3f87ab96-kube-api-access-pxvrx\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.001923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.000656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4caec5d7-6528-4315-9b54-469e3f87ab96-metrics-client-ca\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.001923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.000758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-tls\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.001923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.000862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-grpc-tls\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.001923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.000882 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.102407 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.102324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4caec5d7-6528-4315-9b54-469e3f87ab96-metrics-client-ca\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.102407 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.102396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-tls\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.102599 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.102448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-grpc-tls\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.102662 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.102585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.102739 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.102666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.102805 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.102740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.102805 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.102779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.102895 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.102813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvrx\" (UniqueName: \"kubernetes.io/projected/4caec5d7-6528-4315-9b54-469e3f87ab96-kube-api-access-pxvrx\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.103764 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.103460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4caec5d7-6528-4315-9b54-469e3f87ab96-metrics-client-ca\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.106726 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.105674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-grpc-tls\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.106726 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.106110 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-tls\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.110051 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.110024 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.110193 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.110162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.111458 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.111415 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.111641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.111615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4caec5d7-6528-4315-9b54-469e3f87ab96-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.119557 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.119528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvrx\" (UniqueName: \"kubernetes.io/projected/4caec5d7-6528-4315-9b54-469e3f87ab96-kube-api-access-pxvrx\") pod \"thanos-querier-5f899fbcd8-t77w7\" (UID: \"4caec5d7-6528-4315-9b54-469e3f87ab96\") " pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.152542 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.152521 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:32.250783 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.250758 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:32.254843 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.254812 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:32.396933 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.396909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f899fbcd8-t77w7"] Jul 10 17:12:32.399100 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:32.399075 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caec5d7_6528_4315_9b54_469e3f87ab96.slice/crio-52e0820c9dcc5f2c3a63846d30bf54084fab0dfef29f7254f9f0582ab8d8fb74 WatchSource:0}: Error finding container 52e0820c9dcc5f2c3a63846d30bf54084fab0dfef29f7254f9f0582ab8d8fb74: Status 404 returned error can't find the container with id 52e0820c9dcc5f2c3a63846d30bf54084fab0dfef29f7254f9f0582ab8d8fb74 Jul 10 17:12:32.914765 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.914734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nhfq9" event={"ID":"f68bc723-68a6-4a35-9cf9-4c79895b02cd","Type":"ContainerStarted","Data":"1d965b433f8a068064bd360ed18ab16366354e400239cb83d475f7604de5d100"} Jul 10 17:12:32.915149 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.914794 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nhfq9" event={"ID":"f68bc723-68a6-4a35-9cf9-4c79895b02cd","Type":"ContainerStarted","Data":"2c68698767b622af2c164520cb896bfdfc75d936275210bbb43b1edc21a9f504"} Jul 10 17:12:32.916273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.916249 2580 generic.go:356] "Generic (PLEG): container finished" podID="f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3" containerID="c21e11a339f30aa733f82c952d9f73691282fafdad04e4d61e6bc924977a7aba" exitCode=0 Jul 10 17:12:32.916352 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.916331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3","Type":"ContainerDied","Data":"c21e11a339f30aa733f82c952d9f73691282fafdad04e4d61e6bc924977a7aba"} Jul 10 17:12:32.917549 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.917526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" event={"ID":"4caec5d7-6528-4315-9b54-469e3f87ab96","Type":"ContainerStarted","Data":"52e0820c9dcc5f2c3a63846d30bf54084fab0dfef29f7254f9f0582ab8d8fb74"} Jul 10 17:12:32.939364 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:32.939323 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nhfq9" podStartSLOduration=3.56009648 podStartE2EDuration="4.939306838s" totalImagesPullingTime="1.30570083s" totalInitContainerRuntime="73.509528ms" isStatefulPod=true podCreationTimestamp="2026-07-10 17:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:32.937976415 +0000 UTC m=+79.007173512" watchObservedRunningTime="2026-07-10 17:12:32.939306838 +0000 UTC m=+79.008503941" Jul 10 17:12:33.230555 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.230475 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-645c996df6-cqz6v"] Jul 10 17:12:33.232895 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.232878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.236459 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.236438 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 17:12:33.236459 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.236445 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7hvadilpg635n\"" Jul 10 17:12:33.236647 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.236466 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 17:12:33.236647 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.236471 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 17:12:33.236647 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.236545 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 17:12:33.236647 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.236620 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lbcsd\"" Jul 10 17:12:33.248881 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.248862 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-645c996df6-cqz6v"] Jul 10 17:12:33.313868 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.313842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e1f7706-6282-4803-861d-82728a66b357-secret-metrics-server-client-certs\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.314038 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.313885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e1f7706-6282-4803-861d-82728a66b357-metrics-server-audit-profiles\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.314038 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.313950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e1f7706-6282-4803-861d-82728a66b357-client-ca-bundle\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.314038 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.313981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e1f7706-6282-4803-861d-82728a66b357-audit-log\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.314038 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.314026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e1f7706-6282-4803-861d-82728a66b357-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.314208 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.314056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e1f7706-6282-4803-861d-82728a66b357-secret-metrics-server-tls\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.314208 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.314094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68g6x\" (UniqueName: \"kubernetes.io/projected/7e1f7706-6282-4803-861d-82728a66b357-kube-api-access-68g6x\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.415369 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.415274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e1f7706-6282-4803-861d-82728a66b357-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.415369 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.415346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e1f7706-6282-4803-861d-82728a66b357-secret-metrics-server-tls\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.415369 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.415374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68g6x\" (UniqueName: \"kubernetes.io/projected/7e1f7706-6282-4803-861d-82728a66b357-kube-api-access-68g6x\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.415656 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.415447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e1f7706-6282-4803-861d-82728a66b357-secret-metrics-server-client-certs\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.415656 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.415486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e1f7706-6282-4803-861d-82728a66b357-metrics-server-audit-profiles\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.415656 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.415550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e1f7706-6282-4803-861d-82728a66b357-client-ca-bundle\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.415656 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.415581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e1f7706-6282-4803-861d-82728a66b357-audit-log\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.416185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.416136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e1f7706-6282-4803-861d-82728a66b357-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.416185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.416163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e1f7706-6282-4803-861d-82728a66b357-audit-log\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.416956 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.416912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e1f7706-6282-4803-861d-82728a66b357-metrics-server-audit-profiles\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.418957 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.418754 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e1f7706-6282-4803-861d-82728a66b357-secret-metrics-server-client-certs\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.419208 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.419187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e1f7706-6282-4803-861d-82728a66b357-secret-metrics-server-tls\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.419256 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.419248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e1f7706-6282-4803-861d-82728a66b357-client-ca-bundle\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.426601 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.426576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68g6x\" (UniqueName: \"kubernetes.io/projected/7e1f7706-6282-4803-861d-82728a66b357-kube-api-access-68g6x\") pod \"metrics-server-645c996df6-cqz6v\" (UID: \"7e1f7706-6282-4803-861d-82728a66b357\") " pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.542589 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.542338 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:33.597346 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.597307 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj"] Jul 10 17:12:33.600983 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.600915 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:33.601968 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.601383 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:33.611750 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.611724 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj"] Jul 10 17:12:33.611942 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.611923 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 17:12:33.612186 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.612168 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-stvbj\"" Jul 10 17:12:33.703133 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.703092 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-645c996df6-cqz6v"] Jul 10 17:12:33.707247 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.707147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:33.707247 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:33.707162 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1f7706_6282_4803_861d_82728a66b357.slice/crio-0a49b80011c563330035ef0bb90388dcb2ae8b3abf16151a757afe2332410e7d WatchSource:0}: Error finding container 0a49b80011c563330035ef0bb90388dcb2ae8b3abf16151a757afe2332410e7d: Status 404 returned error can't find the container with id 0a49b80011c563330035ef0bb90388dcb2ae8b3abf16151a757afe2332410e7d Jul 10 17:12:33.721267 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.721240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn8fg\" (UniqueName: \"kubernetes.io/projected/99399577-bfd1-49b8-bb71-6651807c4975-kube-api-access-xn8fg\") pod \"monitoring-plugin-5f68df7f58-bqmrj\" (UID: \"99399577-bfd1-49b8-bb71-6651807c4975\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:33.721366 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.721304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/99399577-bfd1-49b8-bb71-6651807c4975-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-bqmrj\" (UID: \"99399577-bfd1-49b8-bb71-6651807c4975\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:33.821977 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.821896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8fg\" (UniqueName: \"kubernetes.io/projected/99399577-bfd1-49b8-bb71-6651807c4975-kube-api-access-xn8fg\") pod \"monitoring-plugin-5f68df7f58-bqmrj\" (UID: \"99399577-bfd1-49b8-bb71-6651807c4975\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:33.821977 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.821946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/99399577-bfd1-49b8-bb71-6651807c4975-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-bqmrj\" (UID: \"99399577-bfd1-49b8-bb71-6651807c4975\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:33.822144 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:33.822044 2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 10 17:12:33.822144 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:12:33.822097 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99399577-bfd1-49b8-bb71-6651807c4975-monitoring-plugin-cert podName:99399577-bfd1-49b8-bb71-6651807c4975 nodeName:}" failed. No retries permitted until 2026-07-10 17:12:34.322081207 +0000 UTC m=+80.391278285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/99399577-bfd1-49b8-bb71-6651807c4975-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-bqmrj" (UID: "99399577-bfd1-49b8-bb71-6651807c4975") : secret "monitoring-plugin-cert" not found Jul 10 17:12:33.832894 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.832868 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn8fg\" (UniqueName: \"kubernetes.io/projected/99399577-bfd1-49b8-bb71-6651807c4975-kube-api-access-xn8fg\") pod \"monitoring-plugin-5f68df7f58-bqmrj\" (UID: \"99399577-bfd1-49b8-bb71-6651807c4975\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:33.924099 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:33.924061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" event={"ID":"7e1f7706-6282-4803-861d-82728a66b357","Type":"ContainerStarted","Data":"0a49b80011c563330035ef0bb90388dcb2ae8b3abf16151a757afe2332410e7d"} Jul 10 17:12:34.011937 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.011889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:34.084998 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.084923 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-955bdfdbb-8gct9"] Jul 10 17:12:34.088605 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.088581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.091666 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.091522 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 17:12:34.091879 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.091855 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-j8sdl\"" Jul 10 17:12:34.092041 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.091928 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 17:12:34.092041 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.091968 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 17:12:34.092157 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.092125 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 17:12:34.093500 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.093435 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 17:12:34.097945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.097807 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 17:12:34.107027 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.107002 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-955bdfdbb-8gct9"] Jul 10 17:12:34.112201 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.112176 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:34.225727 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.225672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.225915 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.225741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-telemeter-client-tls\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.225915 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.225809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-federate-client-tls\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.225915 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.225848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23106896-b575-4d42-9c9e-1bd865ff1771-metrics-client-ca\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.225915 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.225875 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvvw\" (UniqueName: \"kubernetes.io/projected/23106896-b575-4d42-9c9e-1bd865ff1771-kube-api-access-brvvw\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.226178 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.225983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-secret-telemeter-client\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.226178 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.226046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23106896-b575-4d42-9c9e-1bd865ff1771-telemeter-trusted-ca-bundle\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.226178 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.226082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23106896-b575-4d42-9c9e-1bd865ff1771-serving-certs-ca-bundle\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.328272 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.328232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.328493 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.328291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-telemeter-client-tls\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.328493 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.328313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-federate-client-tls\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.328493 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.328336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23106896-b575-4d42-9c9e-1bd865ff1771-metrics-client-ca\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.328493 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.328363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvvw\" (UniqueName: \"kubernetes.io/projected/23106896-b575-4d42-9c9e-1bd865ff1771-kube-api-access-brvvw\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.328493 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.328403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/99399577-bfd1-49b8-bb71-6651807c4975-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-bqmrj\" (UID: \"99399577-bfd1-49b8-bb71-6651807c4975\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:34.328493 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.328446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-secret-telemeter-client\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.328808 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.328494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23106896-b575-4d42-9c9e-1bd865ff1771-telemeter-trusted-ca-bundle\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.328808 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.328545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23106896-b575-4d42-9c9e-1bd865ff1771-serving-certs-ca-bundle\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.329279 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.329250 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23106896-b575-4d42-9c9e-1bd865ff1771-serving-certs-ca-bundle\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.331063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.331012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23106896-b575-4d42-9c9e-1bd865ff1771-telemeter-trusted-ca-bundle\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.331270 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.331228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23106896-b575-4d42-9c9e-1bd865ff1771-metrics-client-ca\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.331615 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.331578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-telemeter-client-tls\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.331615 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.331587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.332443 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.332421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/99399577-bfd1-49b8-bb71-6651807c4975-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-bqmrj\" (UID: \"99399577-bfd1-49b8-bb71-6651807c4975\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:34.332652 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.332632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-federate-client-tls\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.333200 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.333175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23106896-b575-4d42-9c9e-1bd865ff1771-secret-telemeter-client\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.341731 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.341628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvvw\" (UniqueName: \"kubernetes.io/projected/23106896-b575-4d42-9c9e-1bd865ff1771-kube-api-access-brvvw\") pod \"telemeter-client-955bdfdbb-8gct9\" (UID: \"23106896-b575-4d42-9c9e-1bd865ff1771\") " pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.401646 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.401608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" Jul 10 17:12:34.524228 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.524199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:34.550759 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:34.550726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:35.106735 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.106478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:35.114272 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.113732 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:12:35.118524 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.117299 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.126793 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.124403 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 17:12:35.126793 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.124678 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 17:12:35.126793 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.125052 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-36pegjecl2ige\"" Jul 10 17:12:35.126793 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.125291 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 17:12:35.126793 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.125474 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 17:12:35.136983 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.134794 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 17:12:35.144486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.140908 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 17:12:35.144486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.141163 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 17:12:35.144486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.141362 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 17:12:35.144486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.141547 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fs5x5\"" Jul 10 17:12:35.144486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.141744 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 17:12:35.144486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.141933 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 17:12:35.144486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.142125 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 17:12:35.144486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.143430 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 17:12:35.145060 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.144825 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 17:12:35.145323 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.145135 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 17:12:35.147772 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.146946 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:12:35.212844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.212812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj"] Jul 10 17:12:35.225068 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.221600 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-955bdfdbb-8gct9"] Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.240770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.240815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.240848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.240880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.240916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.240946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qc2\" (UniqueName: \"kubernetes.io/projected/2416a648-be17-44aa-84cc-85a734c114d7-kube-api-access-h5qc2\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.240979 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2416a648-be17-44aa-84cc-85a734c114d7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-web-config\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-config\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.241641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.242527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241226 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2416a648-be17-44aa-84cc-85a734c114d7-config-out\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.242527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.242527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.242527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.242527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.242527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.242527 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.241427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.296674 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.296593 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:35.342193 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342315 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qc2\" (UniqueName: \"kubernetes.io/projected/2416a648-be17-44aa-84cc-85a734c114d7-kube-api-access-h5qc2\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342315 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2416a648-be17-44aa-84cc-85a734c114d7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342315 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342315 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-web-config\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-config\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2416a648-be17-44aa-84cc-85a734c114d7-config-out\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.342538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.342775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.343376 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.343514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.343844 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.343660 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.346333 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.345746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.346333 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.346230 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-config\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.346477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.346446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2416a648-be17-44aa-84cc-85a734c114d7-config-out\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.351361 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.347165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.351361 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.347660 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.351361 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.348156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.351361 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.350685 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.351842 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.351813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.353738 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.353688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.354741 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.354657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.354741 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.354674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2416a648-be17-44aa-84cc-85a734c114d7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.355398 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.355351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2416a648-be17-44aa-84cc-85a734c114d7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.359169 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.359146 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.359526 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.359502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-web-config\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.361161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.361078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.361394 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.361371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2416a648-be17-44aa-84cc-85a734c114d7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.363677 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.363656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qc2\" (UniqueName: \"kubernetes.io/projected/2416a648-be17-44aa-84cc-85a734c114d7-kube-api-access-h5qc2\") pod \"prometheus-k8s-0\" (UID: \"2416a648-be17-44aa-84cc-85a734c114d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.396195 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.396164 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:35.491620 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.491585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:35.770052 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.769984 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:35.871587 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.871564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:35.921965 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.921836 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:12:35.926143 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:12:35.926109 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2416a648_be17_44aa_84cc_85a734c114d7.slice/crio-307d30ec532fcb6345cc3d357d98294242a6f2e4e608e110fbc518bb6dbcebf6 WatchSource:0}: Error finding container 307d30ec532fcb6345cc3d357d98294242a6f2e4e608e110fbc518bb6dbcebf6: Status 404 returned error can't find the container with id 307d30ec532fcb6345cc3d357d98294242a6f2e4e608e110fbc518bb6dbcebf6 Jul 10 17:12:35.935308 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.935200 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2416a648-be17-44aa-84cc-85a734c114d7","Type":"ContainerStarted","Data":"307d30ec532fcb6345cc3d357d98294242a6f2e4e608e110fbc518bb6dbcebf6"} Jul 10 17:12:35.938120 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.937948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" event={"ID":"4caec5d7-6528-4315-9b54-469e3f87ab96","Type":"ContainerStarted","Data":"2d3c00c7c68df8add939d098fe5b5e7c232151865a8fb47c7b917bd873eca98d"} Jul 10 17:12:35.938120 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.938020 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" event={"ID":"4caec5d7-6528-4315-9b54-469e3f87ab96","Type":"ContainerStarted","Data":"e882aa4b581b1707ea644efe7b1e4e7d77ddf9aafcedc8945cf92761de4b1d7a"} Jul 10 17:12:35.938120 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.938063 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" event={"ID":"4caec5d7-6528-4315-9b54-469e3f87ab96","Type":"ContainerStarted","Data":"daa15cb46efb62ec12d357e6178f311eebeb55dc22312fb25b41b3463d657d40"} Jul 10 17:12:35.944283 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.944260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3","Type":"ContainerStarted","Data":"494a768c2ee66ea7bf53404d7522399f54c65598e9372fdfb07e32a79f105d99"} Jul 10 17:12:35.944386 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.944329 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3","Type":"ContainerStarted","Data":"ded68d50369eb755a655f720f29f3860a3232435ffe7be4da38d65f804e8217b"} Jul 10 17:12:35.944386 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.944368 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3","Type":"ContainerStarted","Data":"a0913854445ecb4f17b8ae77409466d39fe10d2e07a3fb4d325d15405b7a04bc"} Jul 10 17:12:35.944488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.944407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3","Type":"ContainerStarted","Data":"79b8b62f089c5d26d7ca5ce709723c10b25d5c29b07b9f2f74c695c129c666cb"} Jul 10 17:12:35.944488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.944447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3","Type":"ContainerStarted","Data":"900f9aa6fcf9b4670ef853256da7a9637c0f764e99afc6cc2c2f711181b3612b"} Jul 10 17:12:35.946054 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.946030 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" event={"ID":"23106896-b575-4d42-9c9e-1bd865ff1771","Type":"ContainerStarted","Data":"491652f65dc98c01294c495f2609de8d63fb8f3bb1b9233a9ab5e472c7352937"} Jul 10 17:12:35.950691 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.950593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" event={"ID":"99399577-bfd1-49b8-bb71-6651807c4975","Type":"ContainerStarted","Data":"d31141befd75611d0f1a6cb2463237599d04704925396f67c86196565df71eae"} Jul 10 17:12:35.953353 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.953023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" event={"ID":"7e1f7706-6282-4803-861d-82728a66b357","Type":"ContainerStarted","Data":"04cfb2a7d9bbc522338f55cd9e0fff6375518efb9e3a72dbc03f70116a0bd0b6"} Jul 10 17:12:35.972309 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.972289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:35.977242 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:35.977194 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" podStartSLOduration=0.871586436 podStartE2EDuration="2.977180731s" totalImagesPullingTime="2.105594295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:35.974887014 +0000 UTC m=+82.044084137" watchObservedRunningTime="2026-07-10 17:12:35.977180731 +0000 UTC m=+82.046377831" Jul 10 17:12:36.071744 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.071656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:36.960926 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.960860 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" event={"ID":"4caec5d7-6528-4315-9b54-469e3f87ab96","Type":"ContainerStarted","Data":"7c4ba1d0d965eb9d8fdca6e53580581278a80d8f645ac21c447949ec310cd00d"} Jul 10 17:12:36.961344 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.960944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" event={"ID":"4caec5d7-6528-4315-9b54-469e3f87ab96","Type":"ContainerStarted","Data":"96f069246f8eb49d81383e9341d82efd94aba5881b1b1c3385a15b26e745e080"} Jul 10 17:12:36.961344 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.960990 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" event={"ID":"4caec5d7-6528-4315-9b54-469e3f87ab96","Type":"ContainerStarted","Data":"105bde73b46908a7d6ef1c88c2c53cb9810461d09ee6e465c30f44cc6e297061"} Jul 10 17:12:36.961344 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.961039 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:36.964981 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.964911 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3","Type":"ContainerStarted","Data":"efefee35cdc8df6824cb3066b0d724bef3367a14c8cc4d70ab81d6cdc6be08fb"} Jul 10 17:12:36.966840 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.966748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" event={"ID":"99399577-bfd1-49b8-bb71-6651807c4975","Type":"ContainerStarted","Data":"85523e930b9a492221a6ff982eab3547660715e4374f9bafba0724f6cb1a7099"} Jul 10 17:12:36.966840 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.966801 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:36.968728 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.968651 2580 generic.go:356] "Generic (PLEG): container finished" podID="2416a648-be17-44aa-84cc-85a734c114d7" containerID="7325656c3d119d1cde03600bed3ae476e9e12f76513437d45a308fcc47d1f150" exitCode=0 Jul 10 17:12:36.968840 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.968783 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2416a648-be17-44aa-84cc-85a734c114d7","Type":"ContainerDied","Data":"7325656c3d119d1cde03600bed3ae476e9e12f76513437d45a308fcc47d1f150"} Jul 10 17:12:36.972659 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.972635 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" Jul 10 17:12:36.990070 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:36.990005 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" podStartSLOduration=2.392381779 podStartE2EDuration="5.98997614s" totalImagesPullingTime="3.597594361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:36.987314313 +0000 UTC m=+83.056511440" watchObservedRunningTime="2026-07-10 17:12:36.98997614 +0000 UTC m=+83.059173241" Jul 10 17:12:37.021633 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:37.021582 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.315702204 podStartE2EDuration="8.021566946s" totalImagesPullingTime="3.685876042s" totalInitContainerRuntime="19.9887ms" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:37.021047719 +0000 UTC m=+83.090244820" watchObservedRunningTime="2026-07-10 17:12:37.021566946 +0000 UTC m=+83.090764089" Jul 10 17:12:37.049960 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:37.049912 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bqmrj" podStartSLOduration=2.412101451 podStartE2EDuration="4.04989673s" totalImagesPullingTime="1.637795279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:37.048224357 +0000 UTC m=+83.117421459" watchObservedRunningTime="2026-07-10 17:12:37.04989673 +0000 UTC m=+83.119093831" Jul 10 17:12:37.976318 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:37.976150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" event={"ID":"23106896-b575-4d42-9c9e-1bd865ff1771","Type":"ContainerStarted","Data":"ebf77145f773ee7346a7d3288e6be349a509a6a544ce1072f1f2869eaecd121c"} Jul 10 17:12:37.976318 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:37.976220 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" event={"ID":"23106896-b575-4d42-9c9e-1bd865ff1771","Type":"ContainerStarted","Data":"8d3542d007b1fc282679190fe074fc00c16d86d166fea7b64b2d755490eab42e"} Jul 10 17:12:37.976318 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:37.976246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" event={"ID":"23106896-b575-4d42-9c9e-1bd865ff1771","Type":"ContainerStarted","Data":"beca000cc431ecf2c4397c83d77cc760355cb304e642c5ad2f47e0dd84ee03e7"} Jul 10 17:12:38.004685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:38.004635 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-955bdfdbb-8gct9" podStartSLOduration=1.538332884 podStartE2EDuration="4.004618184s" totalImagesPullingTime="2.4662853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:38.001874945 +0000 UTC m=+84.071072046" watchObservedRunningTime="2026-07-10 17:12:38.004618184 +0000 UTC m=+84.073815284" Jul 10 17:12:39.769526 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:39.769487 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:39.871068 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:39.871037 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:40.212389 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:40.212352 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:40.313090 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:40.313060 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:41.996207 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:41.996170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2416a648-be17-44aa-84cc-85a734c114d7","Type":"ContainerStarted","Data":"7e31ab887fab8dfed6bbce8eda7e6a3a63572799adfe005ffa7ec955405177d5"} Jul 10 17:12:41.996578 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:41.996240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2416a648-be17-44aa-84cc-85a734c114d7","Type":"ContainerStarted","Data":"b9a6790e24ebe53bd5794dc2246cb5c699aaee62a275c8634dfa84eda553fd87"} Jul 10 17:12:41.996578 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:41.996271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2416a648-be17-44aa-84cc-85a734c114d7","Type":"ContainerStarted","Data":"a73420c2366f2ac240e4b9f77572ef932f15a35c20092e9f00d7640cf12a224e"} Jul 10 17:12:41.996578 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:41.996299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2416a648-be17-44aa-84cc-85a734c114d7","Type":"ContainerStarted","Data":"ea6e7121fa067aca6de1375dbdd0a01823195fb589b64926f3659539aee000b9"} Jul 10 17:12:41.996578 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:41.996325 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2416a648-be17-44aa-84cc-85a734c114d7","Type":"ContainerStarted","Data":"6fc05bad45f3818a7445b2f01dcc69c7db02846bc392e976db3a815465086ef0"} Jul 10 17:12:41.996578 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:41.996359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2416a648-be17-44aa-84cc-85a734c114d7","Type":"ContainerStarted","Data":"cd19e68b23b65aa3efeffe1897a25c5379f0c43e7e6011b5d4019f53cc70e6df"} Jul 10 17:12:42.036026 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:42.035944 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.857160748 podStartE2EDuration="7.035922749s" totalImagesPullingTime="4.150847566s" totalInitContainerRuntime="27.914435ms" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:42.033965182 +0000 UTC m=+88.103162295" watchObservedRunningTime="2026-07-10 17:12:42.035922749 +0000 UTC m=+88.105119850" Jul 10 17:12:42.982948 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:42.982918 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f899fbcd8-t77w7" Jul 10 17:12:44.388763 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:44.388733 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:44.488487 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:44.488436 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:45.492103 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:45.492067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:12:49.019189 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:49.019153 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:49.116776 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:49.116744 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:49.767648 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:49.767616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:49.831925 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:49.831898 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:50.651361 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:50.651318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:12:53.543524 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:53.543496 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:53.543524 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:53.543538 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:12:54.339071 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:54.339041 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k57hc_a75f6837-81ea-42a9-81c8-8808e1e8e44d/serve-healthcheck-canary/0.log" Jul 10 17:12:57.881499 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:57.881466 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-95wjr" Jul 10 17:12:58.977110 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:12:58.977072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:04.014315 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:04.014281 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:04.113644 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:04.113613 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:13.527216 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:13.527178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:13.548051 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:13.548026 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:13:13.551845 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:13.551820 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-645c996df6-cqz6v" Jul 10 17:13:13.651089 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:13.651062 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:13.741796 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:13.741756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:35.491976 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:35.491942 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:13:35.511162 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:35.511138 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:13:36.234251 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:36.234225 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:13:44.967800 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:44.967770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:45.041941 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:45.041905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:56.935926 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:56.935891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:56.994956 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:56.994910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:58.072761 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:58.072690 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:58.172850 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:58.172819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:58.243318 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:58.243273 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:58.795951 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:58.795910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:13:58.896800 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:13:58.896771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:04.017804 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:04.017769 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:04.128208 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:04.128170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:04.246947 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:04.246902 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:09.169506 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:09.169399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:09.268022 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:09.267989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:20.381515 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:20.381477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:24.398465 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:24.398430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:24.499691 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:24.499663 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:37.572825 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:37.572793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:41.735594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:41.735559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:14:41.835993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:14:41.835950 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:08.446210 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:08.446181 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:09.251388 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:09.251347 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:12.056341 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:12.056299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:12.169021 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:12.168991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:14.155751 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:14.155695 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:14.253807 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:14.253768 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:14.621179 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:14.621093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:14.779856 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:14.779821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:16.161805 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.161764 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:16.547112 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.547014 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:16.792155 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.792125 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bgms8"] Jul 10 17:15:16.794486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.794469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" Jul 10 17:15:16.797237 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.797174 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 17:15:16.797237 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.797218 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 17:15:16.798140 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.798122 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xx2c5\"" Jul 10 17:15:16.803772 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.803750 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bgms8"] Jul 10 17:15:16.876947 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.876923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ab45e5a-21c3-490b-ade6-f16d19c5a089-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bgms8\" (UID: \"7ab45e5a-21c3-490b-ade6-f16d19c5a089\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" Jul 10 17:15:16.877043 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.876959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slswn\" (UniqueName: \"kubernetes.io/projected/7ab45e5a-21c3-490b-ade6-f16d19c5a089-kube-api-access-slswn\") pod \"cert-manager-cainjector-559cdd8c87-bgms8\" (UID: \"7ab45e5a-21c3-490b-ade6-f16d19c5a089\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" Jul 10 17:15:16.978169 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.978146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ab45e5a-21c3-490b-ade6-f16d19c5a089-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bgms8\" (UID: \"7ab45e5a-21c3-490b-ade6-f16d19c5a089\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" Jul 10 17:15:16.978261 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.978179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slswn\" (UniqueName: \"kubernetes.io/projected/7ab45e5a-21c3-490b-ade6-f16d19c5a089-kube-api-access-slswn\") pod \"cert-manager-cainjector-559cdd8c87-bgms8\" (UID: \"7ab45e5a-21c3-490b-ade6-f16d19c5a089\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" Jul 10 17:15:16.986428 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.986407 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ab45e5a-21c3-490b-ade6-f16d19c5a089-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bgms8\" (UID: \"7ab45e5a-21c3-490b-ade6-f16d19c5a089\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" Jul 10 17:15:16.986598 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:16.986582 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slswn\" (UniqueName: \"kubernetes.io/projected/7ab45e5a-21c3-490b-ade6-f16d19c5a089-kube-api-access-slswn\") pod \"cert-manager-cainjector-559cdd8c87-bgms8\" (UID: \"7ab45e5a-21c3-490b-ade6-f16d19c5a089\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" Jul 10 17:15:17.115953 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:17.115881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" Jul 10 17:15:17.238674 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:17.238641 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bgms8"] Jul 10 17:15:17.242208 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:15:17.242177 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ab45e5a_21c3_490b_ade6_f16d19c5a089.slice/crio-5833492f69729d8102a06ba899bb86049e3e7ca1e8f9b81d03118a6ec60eb5b9 WatchSource:0}: Error finding container 5833492f69729d8102a06ba899bb86049e3e7ca1e8f9b81d03118a6ec60eb5b9: Status 404 returned error can't find the container with id 5833492f69729d8102a06ba899bb86049e3e7ca1e8f9b81d03118a6ec60eb5b9 Jul 10 17:15:17.612578 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:17.612540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" event={"ID":"7ab45e5a-21c3-490b-ade6-f16d19c5a089","Type":"ContainerStarted","Data":"5833492f69729d8102a06ba899bb86049e3e7ca1e8f9b81d03118a6ec60eb5b9"} Jul 10 17:15:19.048250 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:19.048211 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:19.108429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:19.108360 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:20.630721 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:20.630664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" event={"ID":"7ab45e5a-21c3-490b-ade6-f16d19c5a089","Type":"ContainerStarted","Data":"9b7a3909f6dae732266e4b330c78d6d886961e6e37607ce38338ffc3d94fb6d8"} Jul 10 17:15:20.649883 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:20.649766 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgms8" podStartSLOduration=1.4991605 podStartE2EDuration="4.649744878s" totalImagesPullingTime="3.150584378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:20.647404386 +0000 UTC m=+246.716601487" watchObservedRunningTime="2026-07-10 17:15:20.649744878 +0000 UTC m=+246.718941982" Jul 10 17:15:24.752851 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:24.752810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:25.082429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.082352 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pqsgj"] Jul 10 17:15:25.084841 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.084821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pqsgj" Jul 10 17:15:25.087241 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.087219 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-l79dv\"" Jul 10 17:15:25.095222 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.095203 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pqsgj"] Jul 10 17:15:25.252385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.252353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24b45e56-5be9-4744-90a1-cf47f97099a3-bound-sa-token\") pod \"cert-manager-55b4d954d5-pqsgj\" (UID: \"24b45e56-5be9-4744-90a1-cf47f97099a3\") " pod="cert-manager/cert-manager-55b4d954d5-pqsgj" Jul 10 17:15:25.252542 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.252421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22hdm\" (UniqueName: \"kubernetes.io/projected/24b45e56-5be9-4744-90a1-cf47f97099a3-kube-api-access-22hdm\") pod \"cert-manager-55b4d954d5-pqsgj\" (UID: \"24b45e56-5be9-4744-90a1-cf47f97099a3\") " pod="cert-manager/cert-manager-55b4d954d5-pqsgj" Jul 10 17:15:25.278464 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.278429 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:25.330044 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.330005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:25.353618 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.353549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24b45e56-5be9-4744-90a1-cf47f97099a3-bound-sa-token\") pod \"cert-manager-55b4d954d5-pqsgj\" (UID: \"24b45e56-5be9-4744-90a1-cf47f97099a3\") " pod="cert-manager/cert-manager-55b4d954d5-pqsgj" Jul 10 17:15:25.353618 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.353599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22hdm\" (UniqueName: \"kubernetes.io/projected/24b45e56-5be9-4744-90a1-cf47f97099a3-kube-api-access-22hdm\") pod \"cert-manager-55b4d954d5-pqsgj\" (UID: \"24b45e56-5be9-4744-90a1-cf47f97099a3\") " pod="cert-manager/cert-manager-55b4d954d5-pqsgj" Jul 10 17:15:25.361792 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.361770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24b45e56-5be9-4744-90a1-cf47f97099a3-bound-sa-token\") pod \"cert-manager-55b4d954d5-pqsgj\" (UID: \"24b45e56-5be9-4744-90a1-cf47f97099a3\") " pod="cert-manager/cert-manager-55b4d954d5-pqsgj" Jul 10 17:15:25.361950 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.361930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22hdm\" (UniqueName: \"kubernetes.io/projected/24b45e56-5be9-4744-90a1-cf47f97099a3-kube-api-access-22hdm\") pod \"cert-manager-55b4d954d5-pqsgj\" (UID: \"24b45e56-5be9-4744-90a1-cf47f97099a3\") " pod="cert-manager/cert-manager-55b4d954d5-pqsgj" Jul 10 17:15:25.394288 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.394261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pqsgj" Jul 10 17:15:25.517538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.517490 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pqsgj"] Jul 10 17:15:25.520782 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:15:25.520753 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24b45e56_5be9_4744_90a1_cf47f97099a3.slice/crio-9e95f5a453ebe9f322759b4ac07ff896af089488952e46cfd4ff842b6fad7f10 WatchSource:0}: Error finding container 9e95f5a453ebe9f322759b4ac07ff896af089488952e46cfd4ff842b6fad7f10: Status 404 returned error can't find the container with id 9e95f5a453ebe9f322759b4ac07ff896af089488952e46cfd4ff842b6fad7f10 Jul 10 17:15:25.652771 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.652735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pqsgj" event={"ID":"24b45e56-5be9-4744-90a1-cf47f97099a3","Type":"ContainerStarted","Data":"7277e49764860ea0df0ad63ab62de03676b27a754647ef9a969315c9d1c9b608"} Jul 10 17:15:25.652914 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.652781 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pqsgj" event={"ID":"24b45e56-5be9-4744-90a1-cf47f97099a3","Type":"ContainerStarted","Data":"9e95f5a453ebe9f322759b4ac07ff896af089488952e46cfd4ff842b6fad7f10"} Jul 10 17:15:25.669189 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:25.669147 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pqsgj" podStartSLOduration=0.669133538 podStartE2EDuration="669.133538ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:25.668635982 +0000 UTC m=+251.737833093" watchObservedRunningTime="2026-07-10 17:15:25.669133538 +0000 UTC m=+251.738330637" Jul 10 17:15:30.662258 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:30.662219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:30.762838 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:30.762803 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:37.080121 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:37.080039 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:37.178860 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:37.178832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:39.010235 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:39.010199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:47.246412 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.246378 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:47.298227 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.298191 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv"] Jul 10 17:15:47.301233 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.301211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.304819 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.304791 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 17:15:47.304939 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.304802 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 17:15:47.304939 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.304847 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 17:15:47.304939 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.304914 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6jwqv\"" Jul 10 17:15:47.304939 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.304799 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 17:15:47.304939 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.304935 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:15:47.311930 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.311909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv"] Jul 10 17:15:47.328952 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.328928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:47.338206 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.338181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-manager-config\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.338296 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.338222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9kb\" (UniqueName: \"kubernetes.io/projected/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-kube-api-access-tq9kb\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.338296 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.338265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-cert\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.338296 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.338291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-metrics-cert\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.439588 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.439556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-manager-config\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.439777 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.439592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9kb\" (UniqueName: \"kubernetes.io/projected/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-kube-api-access-tq9kb\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.439777 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.439620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-cert\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.439777 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.439635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-metrics-cert\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.440328 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.440305 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-manager-config\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.442087 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.442060 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-metrics-cert\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.442228 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.442209 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-cert\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.455694 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.455674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9kb\" (UniqueName: \"kubernetes.io/projected/874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc-kube-api-access-tq9kb\") pod \"lws-controller-manager-58d9f8fb8c-ph2dv\" (UID: \"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc\") " pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.612170 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.612108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:47.747341 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:47.747319 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv"] Jul 10 17:15:47.749948 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:15:47.749925 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874b2bcb_91e0_4946_8e5c_5fc3b5a2c0dc.slice/crio-7eb22ff94ffc0428af2311469a10139287d31a12cf223b34be34ae4f9661031b WatchSource:0}: Error finding container 7eb22ff94ffc0428af2311469a10139287d31a12cf223b34be34ae4f9661031b: Status 404 returned error can't find the container with id 7eb22ff94ffc0428af2311469a10139287d31a12cf223b34be34ae4f9661031b Jul 10 17:15:48.748875 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:48.748833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" event={"ID":"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc","Type":"ContainerStarted","Data":"7eb22ff94ffc0428af2311469a10139287d31a12cf223b34be34ae4f9661031b"} Jul 10 17:15:48.906206 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:48.906152 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:50.758881 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:50.758839 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" event={"ID":"874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc","Type":"ContainerStarted","Data":"8158b51055629b09f133536c82cdc5d6ce981bc81dfd5b6e5a9954b30fb7c1d6"} Jul 10 17:15:50.759342 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:50.758976 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:15:50.790348 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:50.790289 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" podStartSLOduration=1.474298377 podStartE2EDuration="3.790267271s" totalImagesPullingTime="2.315968894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:50.787635462 +0000 UTC m=+276.856832563" watchObservedRunningTime="2026-07-10 17:15:50.790267271 +0000 UTC m=+276.859464371" Jul 10 17:15:51.659801 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:51.659772 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:51.756571 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:51.756542 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:52.287962 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.287925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:52.475508 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.475457 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:52.576310 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.576220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:52.634311 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.634280 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z"] Jul 10 17:15:52.637067 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.637047 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.640059 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.640032 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 17:15:52.640184 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.640033 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 17:15:52.640184 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.640076 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 17:15:52.640184 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.640127 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 17:15:52.640505 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.640487 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-znxdl\"" Jul 10 17:15:52.650478 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.650456 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z"] Jul 10 17:15:52.690787 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.690761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6l7\" (UniqueName: \"kubernetes.io/projected/fbead02a-0a11-42c8-a84d-d59c13dbb822-kube-api-access-mr6l7\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.690874 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.690796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbead02a-0a11-42c8-a84d-d59c13dbb822-manifests\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.690874 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.690846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbead02a-0a11-42c8-a84d-d59c13dbb822-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.690963 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.690871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbead02a-0a11-42c8-a84d-d59c13dbb822-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.690963 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.690909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbead02a-0a11-42c8-a84d-d59c13dbb822-tmp\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.751177 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.751153 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:52.791801 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.791778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6l7\" (UniqueName: \"kubernetes.io/projected/fbead02a-0a11-42c8-a84d-d59c13dbb822-kube-api-access-mr6l7\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.791901 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.791808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbead02a-0a11-42c8-a84d-d59c13dbb822-manifests\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.791901 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.791841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbead02a-0a11-42c8-a84d-d59c13dbb822-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.791901 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.791870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbead02a-0a11-42c8-a84d-d59c13dbb822-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.792025 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.791899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbead02a-0a11-42c8-a84d-d59c13dbb822-tmp\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.792246 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.792215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbead02a-0a11-42c8-a84d-d59c13dbb822-manifests\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.794034 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.794007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbead02a-0a11-42c8-a84d-d59c13dbb822-tmp\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.794171 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.794148 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbead02a-0a11-42c8-a84d-d59c13dbb822-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.794220 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.794171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbead02a-0a11-42c8-a84d-d59c13dbb822-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.799277 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.799255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6l7\" (UniqueName: \"kubernetes.io/projected/fbead02a-0a11-42c8-a84d-d59c13dbb822-kube-api-access-mr6l7\") pod \"opendatahub-operator-controller-manager-7b884cfc77-d8l8z\" (UID: \"fbead02a-0a11-42c8-a84d-d59c13dbb822\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.863538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.863484 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:52.947665 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.947631 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:52.955483 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:52.955454 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:53.054941 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:53.054890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:53.083161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:53.083139 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z"] Jul 10 17:15:53.085318 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:15:53.085291 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbead02a_0a11_42c8_a84d_d59c13dbb822.slice/crio-526fedc4b1b409ebc405246a14e40d1234371c606fe60ad3614659802ff00317 WatchSource:0}: Error finding container 526fedc4b1b409ebc405246a14e40d1234371c606fe60ad3614659802ff00317: Status 404 returned error can't find the container with id 526fedc4b1b409ebc405246a14e40d1234371c606fe60ad3614659802ff00317 Jul 10 17:15:53.155167 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:53.155134 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:15:53.771417 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:53.771389 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" event={"ID":"fbead02a-0a11-42c8-a84d-d59c13dbb822","Type":"ContainerStarted","Data":"526fedc4b1b409ebc405246a14e40d1234371c606fe60ad3614659802ff00317"} Jul 10 17:15:56.787695 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:56.787664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" event={"ID":"fbead02a-0a11-42c8-a84d-d59c13dbb822","Type":"ContainerStarted","Data":"c803d91422a550be80485d2874c5494657df2ec12f7735759c1f920027113d83"} Jul 10 17:15:57.793205 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:57.793110 2580 generic.go:356] "Generic (PLEG): container finished" podID="fbead02a-0a11-42c8-a84d-d59c13dbb822" containerID="c803d91422a550be80485d2874c5494657df2ec12f7735759c1f920027113d83" exitCode=0 Jul 10 17:15:57.793205 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:57.793181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" event={"ID":"fbead02a-0a11-42c8-a84d-d59c13dbb822","Type":"ContainerDied","Data":"c803d91422a550be80485d2874c5494657df2ec12f7735759c1f920027113d83"} Jul 10 17:15:58.799415 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:58.799379 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" event={"ID":"fbead02a-0a11-42c8-a84d-d59c13dbb822","Type":"ContainerStarted","Data":"c0e17edc5804f2342255e680e874fa50d3eb7986d78db3bd761f8f8283f5a65c"} Jul 10 17:15:58.799813 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:58.799507 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:15:58.821219 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:15:58.821171 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" podStartSLOduration=2.005154993 podStartE2EDuration="6.821157842s" totalImagesPullingTime="3.805881826s" totalInitContainerRuntime="1.010121023s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:58.819603852 +0000 UTC m=+284.888800951" watchObservedRunningTime="2026-07-10 17:15:58.821157842 +0000 UTC m=+284.890354941" Jul 10 17:16:01.765394 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:01.765364 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58d9f8fb8c-ph2dv" Jul 10 17:16:03.851100 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:03.851062 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:05.059313 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:05.059279 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:09.806677 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:09.806650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-d8l8z" Jul 10 17:16:11.889135 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:11.889099 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:12.015826 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:12.015794 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:14.391009 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:14.390978 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:16:14.398167 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:14.398143 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:16:14.399967 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:14.399950 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 17:16:17.594244 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.594182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:17.681173 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.681148 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-545f68d885-h8rwn"] Jul 10 17:16:17.683933 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.683913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:17.686718 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.686688 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 17:16:17.687016 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.686993 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 17:16:17.687880 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.687860 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 17:16:17.687880 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.687866 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 17:16:17.688030 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.687913 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-42f7l\"" Jul 10 17:16:17.694161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.694142 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545f68d885-h8rwn"] Jul 10 17:16:17.702759 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.702738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4db34a1-8704-4891-a9a6-5ee9e8108291-tls-certs\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:17.702863 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.702773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4db34a1-8704-4891-a9a6-5ee9e8108291-tmp\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:17.702923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.702899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdjsq\" (UniqueName: \"kubernetes.io/projected/d4db34a1-8704-4891-a9a6-5ee9e8108291-kube-api-access-mdjsq\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:17.803444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.803419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdjsq\" (UniqueName: \"kubernetes.io/projected/d4db34a1-8704-4891-a9a6-5ee9e8108291-kube-api-access-mdjsq\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:17.803548 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.803476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4db34a1-8704-4891-a9a6-5ee9e8108291-tls-certs\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:17.803548 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.803503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4db34a1-8704-4891-a9a6-5ee9e8108291-tmp\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:17.803659 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:16:17.803627 2580 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 10 17:16:17.803748 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:16:17.803697 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4db34a1-8704-4891-a9a6-5ee9e8108291-tls-certs podName:d4db34a1-8704-4891-a9a6-5ee9e8108291 nodeName:}" failed. No retries permitted until 2026-07-10 17:16:18.303680136 +0000 UTC m=+304.372877234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d4db34a1-8704-4891-a9a6-5ee9e8108291-tls-certs") pod "kube-auth-proxy-545f68d885-h8rwn" (UID: "d4db34a1-8704-4891-a9a6-5ee9e8108291") : secret "kube-auth-proxy-tls" not found Jul 10 17:16:17.805648 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.805631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4db34a1-8704-4891-a9a6-5ee9e8108291-tmp\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:17.812239 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.812215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdjsq\" (UniqueName: \"kubernetes.io/projected/d4db34a1-8704-4891-a9a6-5ee9e8108291-kube-api-access-mdjsq\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:17.877142 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.877120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:17.937070 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:17.937048 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:18.036626 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.036595 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:18.138930 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.138846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:18.237404 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.237376 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:18.308012 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.307976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4db34a1-8704-4891-a9a6-5ee9e8108291-tls-certs\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:18.310830 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.310809 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4db34a1-8704-4891-a9a6-5ee9e8108291-tls-certs\") pod \"kube-auth-proxy-545f68d885-h8rwn\" (UID: \"d4db34a1-8704-4891-a9a6-5ee9e8108291\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:18.506450 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.506379 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:18.593992 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.593965 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" Jul 10 17:16:18.720846 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.720822 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545f68d885-h8rwn"] Jul 10 17:16:18.723777 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:16:18.723750 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4db34a1_8704_4891_a9a6_5ee9e8108291.slice/crio-555579f1d65fc4d2fe80cbe928aafbfec10fbc639210d74eaf64119d21d3575a WatchSource:0}: Error finding container 555579f1d65fc4d2fe80cbe928aafbfec10fbc639210d74eaf64119d21d3575a: Status 404 returned error can't find the container with id 555579f1d65fc4d2fe80cbe928aafbfec10fbc639210d74eaf64119d21d3575a Jul 10 17:16:18.725612 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.725593 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:16:18.808940 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.808879 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:18.885854 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.885816 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" event={"ID":"d4db34a1-8704-4891-a9a6-5ee9e8108291","Type":"ContainerStarted","Data":"555579f1d65fc4d2fe80cbe928aafbfec10fbc639210d74eaf64119d21d3575a"} Jul 10 17:16:18.901482 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:18.901458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:19.746043 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:19.746001 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:22.915104 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:22.915067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" event={"ID":"d4db34a1-8704-4891-a9a6-5ee9e8108291","Type":"ContainerStarted","Data":"23359b8f0c99583b879ba63d6631123ebe7a21cc3be592e56998a7d38589dcea"} Jul 10 17:16:22.931319 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:22.931266 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-545f68d885-h8rwn" podStartSLOduration=2.387914308 podStartE2EDuration="5.931245352s" totalImagesPullingTime="3.543331044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:16:22.930365245 +0000 UTC m=+308.999562362" watchObservedRunningTime="2026-07-10 17:16:22.931245352 +0000 UTC m=+309.000442453" Jul 10 17:16:25.179050 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:25.179016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:25.280972 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:25.280937 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:25.760945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:25.760908 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:25.908828 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:25.908789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:26.101225 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:26.101146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:29.218975 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:29.218938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:29.316660 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:29.316620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:29.562360 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:29.562273 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:30.144472 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:30.144434 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:30.500726 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:30.500623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:30.605524 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:30.605491 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:16:30.722979 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:16:30.722944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:13.281881 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:13.281788 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:13.376671 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:13.376639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:13.497350 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:13.497315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:14.208073 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:14.208039 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:14.401768 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:14.401733 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:42.508884 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:42.508846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:46.705607 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:46.705570 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:47.255485 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:47.255453 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:49.352040 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:49.352006 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:50.944176 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:50.944135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:52.343688 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:52.343656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:53.897650 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:53.897615 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:54.141504 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:54.141472 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:54.497519 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:54.497481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:56.739735 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:56.739673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:56.840185 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:56.840150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:58.131430 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:58.131387 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:58.233990 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:58.233957 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:58.334036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:58.334003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:58.907398 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:58.907363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:17:59.007204 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:17:59.007172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:00.921566 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:00.921535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:01.022477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:01.022444 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:04.100087 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.100054 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk"] Jul 10 17:18:04.102394 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.102378 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.104858 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.104831 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 17:18:04.104977 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.104928 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2qjm5\"" Jul 10 17:18:04.105050 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.105016 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 17:18:04.105167 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.105149 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 17:18:04.105920 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.105905 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 17:18:04.111772 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.111751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk"] Jul 10 17:18:04.122319 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.122298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:04.222402 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.222371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prsn\" (UniqueName: \"kubernetes.io/projected/808e3317-6dd7-4dd0-bb97-152a393f3cbe-kube-api-access-7prsn\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.222546 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.222415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/808e3317-6dd7-4dd0-bb97-152a393f3cbe-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.222609 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.222548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/808e3317-6dd7-4dd0-bb97-152a393f3cbe-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.323041 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.323007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/808e3317-6dd7-4dd0-bb97-152a393f3cbe-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.323217 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.323054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prsn\" (UniqueName: \"kubernetes.io/projected/808e3317-6dd7-4dd0-bb97-152a393f3cbe-kube-api-access-7prsn\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.323217 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.323082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/808e3317-6dd7-4dd0-bb97-152a393f3cbe-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.323217 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:18:04.323182 2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 10 17:18:04.323371 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:18:04.323248 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/808e3317-6dd7-4dd0-bb97-152a393f3cbe-plugin-serving-cert podName:808e3317-6dd7-4dd0-bb97-152a393f3cbe nodeName:}" failed. No retries permitted until 2026-07-10 17:18:04.823231168 +0000 UTC m=+410.892428246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/808e3317-6dd7-4dd0-bb97-152a393f3cbe-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-cv5nk" (UID: "808e3317-6dd7-4dd0-bb97-152a393f3cbe") : secret "plugin-serving-cert" not found Jul 10 17:18:04.323670 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.323650 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/808e3317-6dd7-4dd0-bb97-152a393f3cbe-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.331769 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.331745 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prsn\" (UniqueName: \"kubernetes.io/projected/808e3317-6dd7-4dd0-bb97-152a393f3cbe-kube-api-access-7prsn\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.662670 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.662641 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:04.828072 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.828039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/808e3317-6dd7-4dd0-bb97-152a393f3cbe-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:04.830516 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:04.830496 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/808e3317-6dd7-4dd0-bb97-152a393f3cbe-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cv5nk\" (UID: \"808e3317-6dd7-4dd0-bb97-152a393f3cbe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:05.011854 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:05.011765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" Jul 10 17:18:05.136555 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:05.136527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk"] Jul 10 17:18:05.138719 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:18:05.138662 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod808e3317_6dd7_4dd0_bb97_152a393f3cbe.slice/crio-312ca249e362dd1ec6d20f45729bb4d3e73db1fe499ff7771ffb09c96b3422b3 WatchSource:0}: Error finding container 312ca249e362dd1ec6d20f45729bb4d3e73db1fe499ff7771ffb09c96b3422b3: Status 404 returned error can't find the container with id 312ca249e362dd1ec6d20f45729bb4d3e73db1fe499ff7771ffb09c96b3422b3 Jul 10 17:18:05.376767 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:05.376735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" event={"ID":"808e3317-6dd7-4dd0-bb97-152a393f3cbe","Type":"ContainerStarted","Data":"312ca249e362dd1ec6d20f45729bb4d3e73db1fe499ff7771ffb09c96b3422b3"} Jul 10 17:18:06.100483 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:06.100449 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:06.202880 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:06.202827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:14.328020 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.327986 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:14.342652 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.342621 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq"] Jul 10 17:18:14.347222 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.347196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:14.350149 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.350126 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q78b9\"" Jul 10 17:18:14.364669 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.364646 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq"] Jul 10 17:18:14.427586 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.427553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:14.521882 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.521841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7pv5\" (UniqueName: \"kubernetes.io/projected/6950a9a4-d073-4826-8856-690f7367a14f-kube-api-access-k7pv5\") pod \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" (UID: \"6950a9a4-d073-4826-8856-690f7367a14f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:14.522079 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.521983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6950a9a4-d073-4826-8856-690f7367a14f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" (UID: \"6950a9a4-d073-4826-8856-690f7367a14f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:14.526964 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.526934 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:14.622815 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.622715 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7pv5\" (UniqueName: \"kubernetes.io/projected/6950a9a4-d073-4826-8856-690f7367a14f-kube-api-access-k7pv5\") pod \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" (UID: \"6950a9a4-d073-4826-8856-690f7367a14f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:14.622997 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.622824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6950a9a4-d073-4826-8856-690f7367a14f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" (UID: \"6950a9a4-d073-4826-8856-690f7367a14f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:14.623280 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.623252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6950a9a4-d073-4826-8856-690f7367a14f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" (UID: \"6950a9a4-d073-4826-8856-690f7367a14f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:14.638839 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.638808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7pv5\" (UniqueName: \"kubernetes.io/projected/6950a9a4-d073-4826-8856-690f7367a14f-kube-api-access-k7pv5\") pod \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" (UID: \"6950a9a4-d073-4826-8856-690f7367a14f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:14.659832 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.659801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:14.752501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.752464 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq"] Jul 10 17:18:14.757961 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.757926 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq"] Jul 10 17:18:14.777091 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.777063 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h"] Jul 10 17:18:14.780866 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.780841 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:14.795466 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.795437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h"] Jul 10 17:18:14.925362 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.925324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6f29e63-f7f4-49cb-973d-544ccc12451b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-pnq5h\" (UID: \"a6f29e63-f7f4-49cb-973d-544ccc12451b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:14.925594 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:14.925378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9t2r\" (UniqueName: \"kubernetes.io/projected/a6f29e63-f7f4-49cb-973d-544ccc12451b-kube-api-access-n9t2r\") pod \"kuadrant-operator-controller-manager-55467d9f7-pnq5h\" (UID: \"a6f29e63-f7f4-49cb-973d-544ccc12451b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:15.026137 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:15.026099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6f29e63-f7f4-49cb-973d-544ccc12451b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-pnq5h\" (UID: \"a6f29e63-f7f4-49cb-973d-544ccc12451b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:15.026331 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:15.026153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9t2r\" (UniqueName: \"kubernetes.io/projected/a6f29e63-f7f4-49cb-973d-544ccc12451b-kube-api-access-n9t2r\") pod \"kuadrant-operator-controller-manager-55467d9f7-pnq5h\" (UID: \"a6f29e63-f7f4-49cb-973d-544ccc12451b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:15.026509 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:15.026486 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6f29e63-f7f4-49cb-973d-544ccc12451b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-pnq5h\" (UID: \"a6f29e63-f7f4-49cb-973d-544ccc12451b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:15.037910 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:15.037882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9t2r\" (UniqueName: \"kubernetes.io/projected/a6f29e63-f7f4-49cb-973d-544ccc12451b-kube-api-access-n9t2r\") pod \"kuadrant-operator-controller-manager-55467d9f7-pnq5h\" (UID: \"a6f29e63-f7f4-49cb-973d-544ccc12451b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:15.096802 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:15.096760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:16.915652 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:16.915620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:17.006017 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:17.005978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:19.893388 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:19.893342 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:19.997535 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:19.997501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:23.696038 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:23.695994 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:23.762157 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:23.762123 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:28.864930 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:28.864905 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h"] Jul 10 17:18:28.866583 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:18:28.866535 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f29e63_f7f4_49cb_973d_544ccc12451b.slice/crio-77e8f18734e7c108b783236a3d7be47e85f9d1f207d7d7c18b4fc993b2a254f0 WatchSource:0}: Error finding container 77e8f18734e7c108b783236a3d7be47e85f9d1f207d7d7c18b4fc993b2a254f0: Status 404 returned error can't find the container with id 77e8f18734e7c108b783236a3d7be47e85f9d1f207d7d7c18b4fc993b2a254f0 Jul 10 17:18:29.502087 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:29.502006 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" event={"ID":"808e3317-6dd7-4dd0-bb97-152a393f3cbe","Type":"ContainerStarted","Data":"98d01ef17bdb8277e6652158cb1ddcf71782d7f6f9be7a9273eed197c6909aad"} Jul 10 17:18:29.503375 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:29.503352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" event={"ID":"a6f29e63-f7f4-49cb-973d-544ccc12451b","Type":"ContainerStarted","Data":"77e8f18734e7c108b783236a3d7be47e85f9d1f207d7d7c18b4fc993b2a254f0"} Jul 10 17:18:29.518546 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:29.518500 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cv5nk" podStartSLOduration=1.518200346 podStartE2EDuration="25.518485439s" totalImagesPullingTime="24.000285093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:29.518299623 +0000 UTC m=+435.587496722" watchObservedRunningTime="2026-07-10 17:18:29.518485439 +0000 UTC m=+435.587682538" Jul 10 17:18:31.689062 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:18:31.689021 2580 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-lpkcq_kuadrant-system_6950a9a4-d073-4826-8856-690f7367a14f_0(a2cd6bbcf26de2c54b59f5f0f2896882b9fffd5c20acaa9eb00a7162e4314467): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-lpkcq to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 10 17:18:31.689369 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:18:31.689100 2580 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-lpkcq_kuadrant-system_6950a9a4-d073-4826-8856-690f7367a14f_0(a2cd6bbcf26de2c54b59f5f0f2896882b9fffd5c20acaa9eb00a7162e4314467): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-lpkcq to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:32.520192 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.520157 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:32.522549 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:18:32.522507 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="6950a9a4-d073-4826-8856-690f7367a14f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:32.539824 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.539800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:32.541945 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:18:32.541920 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="6950a9a4-d073-4826-8856-690f7367a14f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:32.594952 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.594923 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6950a9a4-d073-4826-8856-690f7367a14f-extensions-socket-volume\") pod \"6950a9a4-d073-4826-8856-690f7367a14f\" (UID: \"6950a9a4-d073-4826-8856-690f7367a14f\") " Jul 10 17:18:32.595079 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.594966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7pv5\" (UniqueName: \"kubernetes.io/projected/6950a9a4-d073-4826-8856-690f7367a14f-kube-api-access-k7pv5\") pod \"6950a9a4-d073-4826-8856-690f7367a14f\" (UID: \"6950a9a4-d073-4826-8856-690f7367a14f\") " Jul 10 17:18:32.595166 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.595143 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6950a9a4-d073-4826-8856-690f7367a14f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6950a9a4-d073-4826-8856-690f7367a14f" (UID: "6950a9a4-d073-4826-8856-690f7367a14f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:32.595400 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.595254 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6950a9a4-d073-4826-8856-690f7367a14f-extensions-socket-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:18:32.596994 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.596976 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6950a9a4-d073-4826-8856-690f7367a14f-kube-api-access-k7pv5" (OuterVolumeSpecName: "kube-api-access-k7pv5") pod "6950a9a4-d073-4826-8856-690f7367a14f" (UID: "6950a9a4-d073-4826-8856-690f7367a14f"). InnerVolumeSpecName "kube-api-access-k7pv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:32.696259 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.696226 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7pv5\" (UniqueName: \"kubernetes.io/projected/6950a9a4-d073-4826-8856-690f7367a14f-kube-api-access-k7pv5\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:18:32.898122 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.898095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:32.997858 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:32.997824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:18:33.525255 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:33.525221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:33.527638 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:18:33.527601 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="6950a9a4-d073-4826-8856-690f7367a14f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:33.533416 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:18:33.533385 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="6950a9a4-d073-4826-8856-690f7367a14f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:34.500525 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:18:34.500479 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-lpkcq\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="6950a9a4-d073-4826-8856-690f7367a14f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-lpkcq" Jul 10 17:18:34.501719 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:34.501688 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6950a9a4-d073-4826-8856-690f7367a14f" path="/var/lib/kubelet/pods/6950a9a4-d073-4826-8856-690f7367a14f/volumes" Jul 10 17:18:36.541202 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:36.541164 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" event={"ID":"a6f29e63-f7f4-49cb-973d-544ccc12451b","Type":"ContainerStarted","Data":"2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765"} Jul 10 17:18:36.541563 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:36.541227 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:36.561092 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:36.561046 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" podStartSLOduration=15.730979146 podStartE2EDuration="22.561030972s" totalImagesPullingTime="6.830051826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:36.55881063 +0000 UTC m=+442.628007729" watchObservedRunningTime="2026-07-10 17:18:36.561030972 +0000 UTC m=+442.630228071" Jul 10 17:18:47.547387 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:47.547352 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:18:48.697301 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:18:48.697263 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:00.409805 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.409770 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h"] Jul 10 17:19:00.410198 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.409966 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" podUID="a6f29e63-f7f4-49cb-973d-544ccc12451b" containerName="manager" containerID="cri-o://2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765" gracePeriod=10 Jul 10 17:19:00.647930 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.647904 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:19:00.650970 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.650944 2580 generic.go:356] "Generic (PLEG): container finished" podID="a6f29e63-f7f4-49cb-973d-544ccc12451b" containerID="2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765" exitCode=0 Jul 10 17:19:00.651092 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.650977 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" Jul 10 17:19:00.651092 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.651018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" event={"ID":"a6f29e63-f7f4-49cb-973d-544ccc12451b","Type":"ContainerDied","Data":"2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765"} Jul 10 17:19:00.651092 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.651073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h" event={"ID":"a6f29e63-f7f4-49cb-973d-544ccc12451b","Type":"ContainerDied","Data":"77e8f18734e7c108b783236a3d7be47e85f9d1f207d7d7c18b4fc993b2a254f0"} Jul 10 17:19:00.651194 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.651095 2580 scope.go:117] "RemoveContainer" containerID="2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765" Jul 10 17:19:00.659146 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.659131 2580 scope.go:117] "RemoveContainer" containerID="2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765" Jul 10 17:19:00.659390 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:19:00.659370 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765\": container with ID starting with 2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765 not found: ID does not exist" containerID="2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765" Jul 10 17:19:00.659487 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.659395 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765"} err="failed to get container status \"2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765\": rpc error: code = NotFound desc = could not find container \"2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765\": container with ID starting with 2c3aabe2a70fcf97b666cc03d08e42b54fe278c4f00724ab255f72768dc12765 not found: ID does not exist" Jul 10 17:19:00.742342 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.742267 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6f29e63-f7f4-49cb-973d-544ccc12451b-extensions-socket-volume\") pod \"a6f29e63-f7f4-49cb-973d-544ccc12451b\" (UID: \"a6f29e63-f7f4-49cb-973d-544ccc12451b\") " Jul 10 17:19:00.742342 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.742318 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9t2r\" (UniqueName: \"kubernetes.io/projected/a6f29e63-f7f4-49cb-973d-544ccc12451b-kube-api-access-n9t2r\") pod \"a6f29e63-f7f4-49cb-973d-544ccc12451b\" (UID: \"a6f29e63-f7f4-49cb-973d-544ccc12451b\") " Jul 10 17:19:00.742848 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.742820 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f29e63-f7f4-49cb-973d-544ccc12451b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a6f29e63-f7f4-49cb-973d-544ccc12451b" (UID: "a6f29e63-f7f4-49cb-973d-544ccc12451b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:19:00.744432 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.744413 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f29e63-f7f4-49cb-973d-544ccc12451b-kube-api-access-n9t2r" (OuterVolumeSpecName: "kube-api-access-n9t2r") pod "a6f29e63-f7f4-49cb-973d-544ccc12451b" (UID: "a6f29e63-f7f4-49cb-973d-544ccc12451b"). InnerVolumeSpecName "kube-api-access-n9t2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:19:00.843505 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.843478 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6f29e63-f7f4-49cb-973d-544ccc12451b-extensions-socket-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:19:00.843505 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.843500 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9t2r\" (UniqueName: \"kubernetes.io/projected/a6f29e63-f7f4-49cb-973d-544ccc12451b-kube-api-access-n9t2r\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:19:00.972338 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.972309 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h"] Jul 10 17:19:00.979976 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:00.979954 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-pnq5h"] Jul 10 17:19:01.996665 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:01.996631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:02.501370 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:02.501334 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f29e63-f7f4-49cb-973d-544ccc12451b" path="/var/lib/kubelet/pods/a6f29e63-f7f4-49cb-973d-544ccc12451b/volumes" Jul 10 17:19:03.898659 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:03.898626 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:04.301095 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:04.300994 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:08.423560 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.423526 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:08.529517 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.529478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:08.582860 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.582830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:08.658992 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.658964 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:19:08.659334 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.659321 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f29e63-f7f4-49cb-973d-544ccc12451b" containerName="manager" Jul 10 17:19:08.659375 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.659335 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f29e63-f7f4-49cb-973d-544ccc12451b" containerName="manager" Jul 10 17:19:08.659449 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.659395 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f29e63-f7f4-49cb-973d-544ccc12451b" containerName="manager" Jul 10 17:19:08.661922 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.661904 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:08.665301 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.665281 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 17:19:08.670812 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.670793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:19:08.696180 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.696128 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:19:08.711901 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.711881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84ng\" (UniqueName: \"kubernetes.io/projected/9f3ef799-d4b8-490e-98a0-7a2418b253fa-kube-api-access-k84ng\") pod \"limitador-limitador-54d6d988d5-g7dkh\" (UID: \"9f3ef799-d4b8-490e-98a0-7a2418b253fa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:08.711992 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.711923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f3ef799-d4b8-490e-98a0-7a2418b253fa-config-file\") pod \"limitador-limitador-54d6d988d5-g7dkh\" (UID: \"9f3ef799-d4b8-490e-98a0-7a2418b253fa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:08.812457 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.812429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84ng\" (UniqueName: \"kubernetes.io/projected/9f3ef799-d4b8-490e-98a0-7a2418b253fa-kube-api-access-k84ng\") pod \"limitador-limitador-54d6d988d5-g7dkh\" (UID: \"9f3ef799-d4b8-490e-98a0-7a2418b253fa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:08.812559 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.812543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f3ef799-d4b8-490e-98a0-7a2418b253fa-config-file\") pod \"limitador-limitador-54d6d988d5-g7dkh\" (UID: \"9f3ef799-d4b8-490e-98a0-7a2418b253fa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:08.813175 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.813152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f3ef799-d4b8-490e-98a0-7a2418b253fa-config-file\") pod \"limitador-limitador-54d6d988d5-g7dkh\" (UID: \"9f3ef799-d4b8-490e-98a0-7a2418b253fa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:08.820375 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.820354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84ng\" (UniqueName: \"kubernetes.io/projected/9f3ef799-d4b8-490e-98a0-7a2418b253fa-kube-api-access-k84ng\") pod \"limitador-limitador-54d6d988d5-g7dkh\" (UID: \"9f3ef799-d4b8-490e-98a0-7a2418b253fa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:08.972100 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:08.972035 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:09.100908 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.100884 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:19:09.103369 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:19:09.103341 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3ef799_d4b8_490e_98a0_7a2418b253fa.slice/crio-ef3fce2ea5fc4efb9b934a2ce943439724ad75829cffc638fbd09cf1f2626948 WatchSource:0}: Error finding container ef3fce2ea5fc4efb9b934a2ce943439724ad75829cffc638fbd09cf1f2626948: Status 404 returned error can't find the container with id ef3fce2ea5fc4efb9b934a2ce943439724ad75829cffc638fbd09cf1f2626948 Jul 10 17:19:09.253753 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.253661 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:09.339906 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.339871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:09.340920 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.340902 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-bp4bc"] Jul 10 17:19:09.352081 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.352058 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bp4bc"] Jul 10 17:19:09.352203 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.352131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-bp4bc" Jul 10 17:19:09.355116 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.355097 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ftsm9\"" Jul 10 17:19:09.417603 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.417583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896xc\" (UniqueName: \"kubernetes.io/projected/e327c496-ac4f-40ca-8e4f-2abe1063ce36-kube-api-access-896xc\") pod \"authorino-75d556bd77-bp4bc\" (UID: \"e327c496-ac4f-40ca-8e4f-2abe1063ce36\") " pod="kuadrant-system/authorino-75d556bd77-bp4bc" Jul 10 17:19:09.518091 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.518038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896xc\" (UniqueName: \"kubernetes.io/projected/e327c496-ac4f-40ca-8e4f-2abe1063ce36-kube-api-access-896xc\") pod \"authorino-75d556bd77-bp4bc\" (UID: \"e327c496-ac4f-40ca-8e4f-2abe1063ce36\") " pod="kuadrant-system/authorino-75d556bd77-bp4bc" Jul 10 17:19:09.529252 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.529230 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-896xc\" (UniqueName: \"kubernetes.io/projected/e327c496-ac4f-40ca-8e4f-2abe1063ce36-kube-api-access-896xc\") pod \"authorino-75d556bd77-bp4bc\" (UID: \"e327c496-ac4f-40ca-8e4f-2abe1063ce36\") " pod="kuadrant-system/authorino-75d556bd77-bp4bc" Jul 10 17:19:09.661817 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.661799 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-bp4bc" Jul 10 17:19:09.692361 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.692336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" event={"ID":"9f3ef799-d4b8-490e-98a0-7a2418b253fa","Type":"ContainerStarted","Data":"ef3fce2ea5fc4efb9b934a2ce943439724ad75829cffc638fbd09cf1f2626948"} Jul 10 17:19:09.806020 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:09.805997 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bp4bc"] Jul 10 17:19:09.807800 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:19:09.807776 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode327c496_ac4f_40ca_8e4f_2abe1063ce36.slice/crio-0ace13697cf8910aadd1c0b65f365b6788ba20996a2a72c3e86cf9c0699920fd WatchSource:0}: Error finding container 0ace13697cf8910aadd1c0b65f365b6788ba20996a2a72c3e86cf9c0699920fd: Status 404 returned error can't find the container with id 0ace13697cf8910aadd1c0b65f365b6788ba20996a2a72c3e86cf9c0699920fd Jul 10 17:19:10.700743 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:10.700688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-bp4bc" event={"ID":"e327c496-ac4f-40ca-8e4f-2abe1063ce36","Type":"ContainerStarted","Data":"0ace13697cf8910aadd1c0b65f365b6788ba20996a2a72c3e86cf9c0699920fd"} Jul 10 17:19:14.720921 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:14.720827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" event={"ID":"9f3ef799-d4b8-490e-98a0-7a2418b253fa","Type":"ContainerStarted","Data":"069919cffd0360112a0af70a5fd07c6e69f8c8d11032069a7918baae070e1756"} Jul 10 17:19:14.721346 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:14.720926 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:14.722262 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:14.722242 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-bp4bc" event={"ID":"e327c496-ac4f-40ca-8e4f-2abe1063ce36","Type":"ContainerStarted","Data":"15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d"} Jul 10 17:19:14.736964 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:14.736923 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" podStartSLOduration=2.101934207 podStartE2EDuration="6.73691201s" totalImagesPullingTime="4.634977803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:19:14.735408154 +0000 UTC m=+480.804605254" watchObservedRunningTime="2026-07-10 17:19:14.73691201 +0000 UTC m=+480.806109109" Jul 10 17:19:14.753020 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:14.752975 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-bp4bc" podStartSLOduration=1.819148547 podStartE2EDuration="5.752963446s" totalImagesPullingTime="3.933814899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:19:14.752141536 +0000 UTC m=+480.821338636" watchObservedRunningTime="2026-07-10 17:19:14.752963446 +0000 UTC m=+480.822160546" Jul 10 17:19:25.727478 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:25.727445 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g7dkh" Jul 10 17:19:29.618978 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:29.618941 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:29.722889 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:29.722858 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:30.065042 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:30.065011 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:41.951665 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:41.951623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:42.174211 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.174174 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dm6x9"] Jul 10 17:19:42.181902 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.181876 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dm6x9"] Jul 10 17:19:42.182053 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.181957 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dm6x9" Jul 10 17:19:42.305181 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.305095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mg8\" (UniqueName: \"kubernetes.io/projected/ec0009a7-e539-4b71-8d30-71bffa7bc5bc-kube-api-access-s6mg8\") pod \"authorino-5d4b67c446-dm6x9\" (UID: \"ec0009a7-e539-4b71-8d30-71bffa7bc5bc\") " pod="kuadrant-system/authorino-5d4b67c446-dm6x9" Jul 10 17:19:42.355881 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.355850 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:42.382791 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.382763 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dm6x9"] Jul 10 17:19:42.382985 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:19:42.382966 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s6mg8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-dm6x9" podUID="ec0009a7-e539-4b71-8d30-71bffa7bc5bc" Jul 10 17:19:42.402093 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.402069 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59bcbfd7ff-tr9ks"] Jul 10 17:19:42.405580 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.405563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59bcbfd7ff-tr9ks" Jul 10 17:19:42.405789 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.405767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mg8\" (UniqueName: \"kubernetes.io/projected/ec0009a7-e539-4b71-8d30-71bffa7bc5bc-kube-api-access-s6mg8\") pod \"authorino-5d4b67c446-dm6x9\" (UID: \"ec0009a7-e539-4b71-8d30-71bffa7bc5bc\") " pod="kuadrant-system/authorino-5d4b67c446-dm6x9" Jul 10 17:19:42.412804 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.412784 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59bcbfd7ff-tr9ks"] Jul 10 17:19:42.415509 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.415487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mg8\" (UniqueName: \"kubernetes.io/projected/ec0009a7-e539-4b71-8d30-71bffa7bc5bc-kube-api-access-s6mg8\") pod \"authorino-5d4b67c446-dm6x9\" (UID: \"ec0009a7-e539-4b71-8d30-71bffa7bc5bc\") " pod="kuadrant-system/authorino-5d4b67c446-dm6x9" Jul 10 17:19:42.506530 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.506497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvlkt\" (UniqueName: \"kubernetes.io/projected/81e84d01-dbe1-431c-83ea-3e12c490221e-kube-api-access-wvlkt\") pod \"authorino-59bcbfd7ff-tr9ks\" (UID: \"81e84d01-dbe1-431c-83ea-3e12c490221e\") " pod="kuadrant-system/authorino-59bcbfd7ff-tr9ks" Jul 10 17:19:42.539407 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.539380 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59bcbfd7ff-tr9ks"] Jul 10 17:19:42.539587 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:19:42.539570 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wvlkt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-59bcbfd7ff-tr9ks" podUID="81e84d01-dbe1-431c-83ea-3e12c490221e" Jul 10 17:19:42.567272 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.567210 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5bc58b6dc-5s8lr"] Jul 10 17:19:42.570841 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.570822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:19:42.573013 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.572995 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 17:19:42.577781 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.577760 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bc58b6dc-5s8lr"] Jul 10 17:19:42.607091 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.607060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvlkt\" (UniqueName: \"kubernetes.io/projected/81e84d01-dbe1-431c-83ea-3e12c490221e-kube-api-access-wvlkt\") pod \"authorino-59bcbfd7ff-tr9ks\" (UID: \"81e84d01-dbe1-431c-83ea-3e12c490221e\") " pod="kuadrant-system/authorino-59bcbfd7ff-tr9ks" Jul 10 17:19:42.615316 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.615298 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvlkt\" (UniqueName: \"kubernetes.io/projected/81e84d01-dbe1-431c-83ea-3e12c490221e-kube-api-access-wvlkt\") pod \"authorino-59bcbfd7ff-tr9ks\" (UID: \"81e84d01-dbe1-431c-83ea-3e12c490221e\") " pod="kuadrant-system/authorino-59bcbfd7ff-tr9ks" Jul 10 17:19:42.707851 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.707816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5kl\" (UniqueName: \"kubernetes.io/projected/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-kube-api-access-dg5kl\") pod \"authorino-5bc58b6dc-5s8lr\" (UID: \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\") " pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:19:42.708026 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.707900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-tls-cert\") pod \"authorino-5bc58b6dc-5s8lr\" (UID: \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\") " pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:19:42.808604 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.808573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5kl\" (UniqueName: \"kubernetes.io/projected/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-kube-api-access-dg5kl\") pod \"authorino-5bc58b6dc-5s8lr\" (UID: \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\") " pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:19:42.808818 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.808670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-tls-cert\") pod \"authorino-5bc58b6dc-5s8lr\" (UID: \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\") " pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:19:42.811031 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.811010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-tls-cert\") pod \"authorino-5bc58b6dc-5s8lr\" (UID: \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\") " pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:19:42.816254 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.816233 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5kl\" (UniqueName: \"kubernetes.io/projected/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-kube-api-access-dg5kl\") pod \"authorino-5bc58b6dc-5s8lr\" (UID: \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\") " pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:19:42.847138 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.847082 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59bcbfd7ff-tr9ks" Jul 10 17:19:42.847231 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.847084 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dm6x9" Jul 10 17:19:42.854204 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.854186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59bcbfd7ff-tr9ks" Jul 10 17:19:42.857285 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.857268 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dm6x9" Jul 10 17:19:42.880399 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:42.880378 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:19:43.004182 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.004026 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bc58b6dc-5s8lr"] Jul 10 17:19:43.006410 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:19:43.006386 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6101b6c6_9caf_41b7_ab2a_8a816d05e0bd.slice/crio-2a09daa6bfaf36c22252cd42269d3668258b34556ba2d558b3963c9db8a6f0f0 WatchSource:0}: Error finding container 2a09daa6bfaf36c22252cd42269d3668258b34556ba2d558b3963c9db8a6f0f0: Status 404 returned error can't find the container with id 2a09daa6bfaf36c22252cd42269d3668258b34556ba2d558b3963c9db8a6f0f0 Jul 10 17:19:43.010394 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.010375 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvlkt\" (UniqueName: \"kubernetes.io/projected/81e84d01-dbe1-431c-83ea-3e12c490221e-kube-api-access-wvlkt\") pod \"81e84d01-dbe1-431c-83ea-3e12c490221e\" (UID: \"81e84d01-dbe1-431c-83ea-3e12c490221e\") " Jul 10 17:19:43.010456 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.010405 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6mg8\" (UniqueName: \"kubernetes.io/projected/ec0009a7-e539-4b71-8d30-71bffa7bc5bc-kube-api-access-s6mg8\") pod \"ec0009a7-e539-4b71-8d30-71bffa7bc5bc\" (UID: \"ec0009a7-e539-4b71-8d30-71bffa7bc5bc\") " Jul 10 17:19:43.012317 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.012294 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0009a7-e539-4b71-8d30-71bffa7bc5bc-kube-api-access-s6mg8" (OuterVolumeSpecName: "kube-api-access-s6mg8") pod "ec0009a7-e539-4b71-8d30-71bffa7bc5bc" (UID: "ec0009a7-e539-4b71-8d30-71bffa7bc5bc"). InnerVolumeSpecName "kube-api-access-s6mg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:19:43.012402 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.012380 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e84d01-dbe1-431c-83ea-3e12c490221e-kube-api-access-wvlkt" (OuterVolumeSpecName: "kube-api-access-wvlkt") pod "81e84d01-dbe1-431c-83ea-3e12c490221e" (UID: "81e84d01-dbe1-431c-83ea-3e12c490221e"). InnerVolumeSpecName "kube-api-access-wvlkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:19:43.112046 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.111991 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvlkt\" (UniqueName: \"kubernetes.io/projected/81e84d01-dbe1-431c-83ea-3e12c490221e-kube-api-access-wvlkt\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:19:43.112046 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.112015 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mg8\" (UniqueName: \"kubernetes.io/projected/ec0009a7-e539-4b71-8d30-71bffa7bc5bc-kube-api-access-s6mg8\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:19:43.853114 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.853085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dm6x9" Jul 10 17:19:43.853311 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.853116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" event={"ID":"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd","Type":"ContainerStarted","Data":"65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4"} Jul 10 17:19:43.853311 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.853162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" event={"ID":"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd","Type":"ContainerStarted","Data":"2a09daa6bfaf36c22252cd42269d3668258b34556ba2d558b3963c9db8a6f0f0"} Jul 10 17:19:43.853311 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.853266 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59bcbfd7ff-tr9ks" Jul 10 17:19:43.868304 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.868248 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" podStartSLOduration=1.522144999 podStartE2EDuration="1.868230108s" totalImagesPullingTime="346.085109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:19:43.867564958 +0000 UTC m=+509.936762057" watchObservedRunningTime="2026-07-10 17:19:43.868230108 +0000 UTC m=+509.937427209" Jul 10 17:19:43.894242 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.894220 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dm6x9"] Jul 10 17:19:43.901441 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.901418 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bp4bc"] Jul 10 17:19:43.901587 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.901569 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-bp4bc" podUID="e327c496-ac4f-40ca-8e4f-2abe1063ce36" containerName="authorino" containerID="cri-o://15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d" gracePeriod=30 Jul 10 17:19:43.903079 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.903060 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dm6x9"] Jul 10 17:19:43.927253 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.927229 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59bcbfd7ff-tr9ks"] Jul 10 17:19:43.932117 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:43.932092 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59bcbfd7ff-tr9ks"] Jul 10 17:19:44.152079 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.152051 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-bp4bc" Jul 10 17:19:44.324248 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.324215 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-896xc\" (UniqueName: \"kubernetes.io/projected/e327c496-ac4f-40ca-8e4f-2abe1063ce36-kube-api-access-896xc\") pod \"e327c496-ac4f-40ca-8e4f-2abe1063ce36\" (UID: \"e327c496-ac4f-40ca-8e4f-2abe1063ce36\") " Jul 10 17:19:44.326275 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.326246 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e327c496-ac4f-40ca-8e4f-2abe1063ce36-kube-api-access-896xc" (OuterVolumeSpecName: "kube-api-access-896xc") pod "e327c496-ac4f-40ca-8e4f-2abe1063ce36" (UID: "e327c496-ac4f-40ca-8e4f-2abe1063ce36"). InnerVolumeSpecName "kube-api-access-896xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:19:44.425066 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.424994 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-896xc\" (UniqueName: \"kubernetes.io/projected/e327c496-ac4f-40ca-8e4f-2abe1063ce36-kube-api-access-896xc\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:19:44.502061 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.502033 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e84d01-dbe1-431c-83ea-3e12c490221e" path="/var/lib/kubelet/pods/81e84d01-dbe1-431c-83ea-3e12c490221e/volumes" Jul 10 17:19:44.502268 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.502255 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec0009a7-e539-4b71-8d30-71bffa7bc5bc" path="/var/lib/kubelet/pods/ec0009a7-e539-4b71-8d30-71bffa7bc5bc/volumes" Jul 10 17:19:44.861050 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.860958 2580 generic.go:356] "Generic (PLEG): container finished" podID="e327c496-ac4f-40ca-8e4f-2abe1063ce36" containerID="15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d" exitCode=0 Jul 10 17:19:44.861050 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.861009 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-bp4bc" Jul 10 17:19:44.861251 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.861043 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-bp4bc" event={"ID":"e327c496-ac4f-40ca-8e4f-2abe1063ce36","Type":"ContainerDied","Data":"15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d"} Jul 10 17:19:44.861251 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.861086 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-bp4bc" event={"ID":"e327c496-ac4f-40ca-8e4f-2abe1063ce36","Type":"ContainerDied","Data":"0ace13697cf8910aadd1c0b65f365b6788ba20996a2a72c3e86cf9c0699920fd"} Jul 10 17:19:44.861251 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.861105 2580 scope.go:117] "RemoveContainer" containerID="15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d" Jul 10 17:19:44.870394 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.870375 2580 scope.go:117] "RemoveContainer" containerID="15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d" Jul 10 17:19:44.870639 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:19:44.870617 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d\": container with ID starting with 15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d not found: ID does not exist" containerID="15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d" Jul 10 17:19:44.870696 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.870648 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d"} err="failed to get container status \"15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d\": rpc error: code = NotFound desc = could not find container \"15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d\": container with ID starting with 15a302c14e3d0fcb06592bbafb00e36e05d5dd05d9d7ac6988dd9a30df3a5c9d not found: ID does not exist" Jul 10 17:19:44.876163 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.876137 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bp4bc"] Jul 10 17:19:44.878530 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:44.878509 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bp4bc"] Jul 10 17:19:46.501875 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:46.501842 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e327c496-ac4f-40ca-8e4f-2abe1063ce36" path="/var/lib/kubelet/pods/e327c496-ac4f-40ca-8e4f-2abe1063ce36/volumes" Jul 10 17:19:48.028641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:48.028610 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:48.094278 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:48.094230 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:48.151103 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:48.151079 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:48.356950 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:48.356877 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:51.163434 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:51.163391 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:19:51.262887 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:19:51.262831 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:04.975216 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:04.975165 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:18.106664 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:18.106623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:22.595074 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:22.595039 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:22.688036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:22.688008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:23.170273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:23.170233 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:23.678822 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:23.678782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:23.783242 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:23.783212 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:23.944653 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:23.944587 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:20:24.037683 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:24.037649 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:33.619824 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:33.619792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:33.705350 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:33.705319 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:44.165322 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:44.165290 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:44.655948 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:44.655910 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:20:58.655443 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:58.655411 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:20:59.262436 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:20:59.262402 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:21:07.576363 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:07.576332 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:08.248012 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:08.247979 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:21:14.424342 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:14.424315 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:21:14.433755 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:14.433725 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:21:17.383001 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:17.382967 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:19.261988 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:19.261929 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:21:21.546910 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.546879 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2"] Jul 10 17:21:21.547266 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.547226 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e327c496-ac4f-40ca-8e4f-2abe1063ce36" containerName="authorino" Jul 10 17:21:21.547266 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.547237 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e327c496-ac4f-40ca-8e4f-2abe1063ce36" containerName="authorino" Jul 10 17:21:21.547340 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.547316 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e327c496-ac4f-40ca-8e4f-2abe1063ce36" containerName="authorino" Jul 10 17:21:21.551777 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.551747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.555980 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.555959 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 17:21:21.556800 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.556781 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 17:21:21.556893 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.556787 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 17:21:21.556893 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.556789 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5vqx5\"" Jul 10 17:21:21.573569 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.573547 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2"] Jul 10 17:21:21.587263 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.587240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.587368 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.587294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.587368 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.587311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41fcfde-8f83-4209-b40a-b4aa21fa930d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.587470 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.587362 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwzgh\" (UniqueName: \"kubernetes.io/projected/d41fcfde-8f83-4209-b40a-b4aa21fa930d-kube-api-access-vwzgh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.587470 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.587403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.587552 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.587473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.619532 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.619507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:21.687972 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.687936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.688152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.687994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.688152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.688105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41fcfde-8f83-4209-b40a-b4aa21fa930d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.688152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.688134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwzgh\" (UniqueName: \"kubernetes.io/projected/d41fcfde-8f83-4209-b40a-b4aa21fa930d-kube-api-access-vwzgh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.688323 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.688155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.688323 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.688232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.688461 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.688439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.688524 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.688497 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.688565 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.688548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.690234 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.690211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41fcfde-8f83-4209-b40a-b4aa21fa930d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.690489 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.690473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41fcfde-8f83-4209-b40a-b4aa21fa930d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.696548 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.696524 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwzgh\" (UniqueName: \"kubernetes.io/projected/d41fcfde-8f83-4209-b40a-b4aa21fa930d-kube-api-access-vwzgh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sb8z2\" (UID: \"d41fcfde-8f83-4209-b40a-b4aa21fa930d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.863278 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.863204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:21.993800 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.993776 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2"] Jul 10 17:21:21.995645 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:21:21.995616 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41fcfde_8f83_4209_b40a_b4aa21fa930d.slice/crio-099689dc69fde82607ae90a49ab06cd5e6d9415b3db039bf30df2bbc3f11105e WatchSource:0}: Error finding container 099689dc69fde82607ae90a49ab06cd5e6d9415b3db039bf30df2bbc3f11105e: Status 404 returned error can't find the container with id 099689dc69fde82607ae90a49ab06cd5e6d9415b3db039bf30df2bbc3f11105e Jul 10 17:21:21.997549 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:21.997529 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:21:22.307204 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:22.307157 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" event={"ID":"d41fcfde-8f83-4209-b40a-b4aa21fa930d","Type":"ContainerStarted","Data":"099689dc69fde82607ae90a49ab06cd5e6d9415b3db039bf30df2bbc3f11105e"} Jul 10 17:21:23.314784 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.314675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" event={"ID":"d41fcfde-8f83-4209-b40a-b4aa21fa930d","Type":"ContainerStarted","Data":"c136067f0e6410123ab2ebc0e93b0467876afcc84e88880b78d239dc4d122c5a"} Jul 10 17:21:23.333355 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.333299 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" podStartSLOduration=1.307435175 podStartE2EDuration="2.333279726s" totalImagesPullingTime="1.025844551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:21:23.331533799 +0000 UTC m=+609.400730900" watchObservedRunningTime="2026-07-10 17:21:23.333279726 +0000 UTC m=+609.402476827" Jul 10 17:21:23.746121 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.746085 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn"] Jul 10 17:21:23.750267 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.750244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:23.752979 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.752767 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 17:21:23.763027 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.763007 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn"] Jul 10 17:21:23.780658 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.780635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:23.909245 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.909206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:23.909398 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.909246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:23.909398 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.909297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:23.909398 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.909315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf26c\" (UniqueName: \"kubernetes.io/projected/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-kube-api-access-bf26c\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:23.909504 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.909428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:23.909504 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:23.909455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.003637 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.003567 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:21:24.009852 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.009831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.009918 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.009860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.009918 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.009883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.009918 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.009910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf26c\" (UniqueName: \"kubernetes.io/projected/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-kube-api-access-bf26c\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.010032 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.009989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.010032 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.010011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.010203 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.010182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.010327 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.010311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.010383 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.010346 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.012201 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.012177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.012339 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.012324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.018565 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.018545 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf26c\" (UniqueName: \"kubernetes.io/projected/8d21842c-61ed-4015-aa8b-9d5f69a3fd4c-kube-api-access-bf26c\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn\" (UID: \"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.060473 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.060450 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:24.200043 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.200018 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn"] Jul 10 17:21:24.202490 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:21:24.202462 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d21842c_61ed_4015_aa8b_9d5f69a3fd4c.slice/crio-12e0bd0bf4cd0c468ab646d193252a8920215408ff8d8d9973b6438857a6cf93 WatchSource:0}: Error finding container 12e0bd0bf4cd0c468ab646d193252a8920215408ff8d8d9973b6438857a6cf93: Status 404 returned error can't find the container with id 12e0bd0bf4cd0c468ab646d193252a8920215408ff8d8d9973b6438857a6cf93 Jul 10 17:21:24.320117 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:24.320031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" event={"ID":"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c","Type":"ContainerStarted","Data":"12e0bd0bf4cd0c468ab646d193252a8920215408ff8d8d9973b6438857a6cf93"} Jul 10 17:21:25.325629 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:25.325594 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" event={"ID":"8d21842c-61ed-4015-aa8b-9d5f69a3fd4c","Type":"ContainerStarted","Data":"3cccbcf8e6754e08f0a97df6f08ad73cb1ca889b7080cd2c67c8f7c507cb0280"} Jul 10 17:21:25.342534 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:25.342484 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" podStartSLOduration=2.148830282 podStartE2EDuration="2.342469762s" totalImagesPullingTime="193.63948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:21:25.340413371 +0000 UTC m=+611.409610471" watchObservedRunningTime="2026-07-10 17:21:25.342469762 +0000 UTC m=+611.411666861" Jul 10 17:21:28.357768 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:28.357733 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:21:31.864011 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:31.863971 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:31.864384 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:31.864056 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:31.876244 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:31.876218 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:32.372161 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:32.372134 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sb8z2" Jul 10 17:21:34.061336 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:34.061302 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:34.061758 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:34.061349 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:34.073784 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:34.073756 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:21:34.381332 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:21:34.381300 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn" Jul 10 17:22:28.475358 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.475322 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-764b77c78b-577vn"] Jul 10 17:22:28.478803 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.478781 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-764b77c78b-577vn" Jul 10 17:22:28.488882 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.488861 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-764b77c78b-577vn"] Jul 10 17:22:28.592091 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.592068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19426143-0472-46ad-b7e6-1e09bb828f86-tls-cert\") pod \"authorino-764b77c78b-577vn\" (UID: \"19426143-0472-46ad-b7e6-1e09bb828f86\") " pod="kuadrant-system/authorino-764b77c78b-577vn" Jul 10 17:22:28.592198 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.592100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475sj\" (UniqueName: \"kubernetes.io/projected/19426143-0472-46ad-b7e6-1e09bb828f86-kube-api-access-475sj\") pod \"authorino-764b77c78b-577vn\" (UID: \"19426143-0472-46ad-b7e6-1e09bb828f86\") " pod="kuadrant-system/authorino-764b77c78b-577vn" Jul 10 17:22:28.693359 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.693336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19426143-0472-46ad-b7e6-1e09bb828f86-tls-cert\") pod \"authorino-764b77c78b-577vn\" (UID: \"19426143-0472-46ad-b7e6-1e09bb828f86\") " pod="kuadrant-system/authorino-764b77c78b-577vn" Jul 10 17:22:28.693462 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.693369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475sj\" (UniqueName: \"kubernetes.io/projected/19426143-0472-46ad-b7e6-1e09bb828f86-kube-api-access-475sj\") pod \"authorino-764b77c78b-577vn\" (UID: \"19426143-0472-46ad-b7e6-1e09bb828f86\") " pod="kuadrant-system/authorino-764b77c78b-577vn" Jul 10 17:22:28.695823 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.695799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19426143-0472-46ad-b7e6-1e09bb828f86-tls-cert\") pod \"authorino-764b77c78b-577vn\" (UID: \"19426143-0472-46ad-b7e6-1e09bb828f86\") " pod="kuadrant-system/authorino-764b77c78b-577vn" Jul 10 17:22:28.702563 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.702533 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-475sj\" (UniqueName: \"kubernetes.io/projected/19426143-0472-46ad-b7e6-1e09bb828f86-kube-api-access-475sj\") pod \"authorino-764b77c78b-577vn\" (UID: \"19426143-0472-46ad-b7e6-1e09bb828f86\") " pod="kuadrant-system/authorino-764b77c78b-577vn" Jul 10 17:22:28.788280 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.788227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-764b77c78b-577vn" Jul 10 17:22:28.860348 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.860318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:28.929089 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.926085 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-764b77c78b-577vn"] Jul 10 17:22:28.929236 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:28.929142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:28.929364 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:22:28.929330 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19426143_0472_46ad_b7e6_1e09bb828f86.slice/crio-c65bc828be32aa33adb2a240cd026e7e12b4676f5d8b94d99985d2a6d9ca404f WatchSource:0}: Error finding container c65bc828be32aa33adb2a240cd026e7e12b4676f5d8b94d99985d2a6d9ca404f: Status 404 returned error can't find the container with id c65bc828be32aa33adb2a240cd026e7e12b4676f5d8b94d99985d2a6d9ca404f Jul 10 17:22:29.624002 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:29.623956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-764b77c78b-577vn" event={"ID":"19426143-0472-46ad-b7e6-1e09bb828f86","Type":"ContainerStarted","Data":"bf2cabf19f51714eba350639f019b81a51a5fa50befdbbe343dee3543562cbd8"} Jul 10 17:22:29.624002 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:29.624007 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-764b77c78b-577vn" event={"ID":"19426143-0472-46ad-b7e6-1e09bb828f86","Type":"ContainerStarted","Data":"c65bc828be32aa33adb2a240cd026e7e12b4676f5d8b94d99985d2a6d9ca404f"} Jul 10 17:22:29.639611 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:29.639554 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-764b77c78b-577vn" podStartSLOduration=1.1034403799999999 podStartE2EDuration="1.639534086s" totalImagesPullingTime="536.093706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:22:29.63906548 +0000 UTC m=+675.708262593" watchObservedRunningTime="2026-07-10 17:22:29.639534086 +0000 UTC m=+675.708731187" Jul 10 17:22:29.666751 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:29.666716 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bc58b6dc-5s8lr"] Jul 10 17:22:29.666907 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:29.666878 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" podUID="6101b6c6-9caf-41b7-ab2a-8a816d05e0bd" containerName="authorino" containerID="cri-o://65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4" gracePeriod=30 Jul 10 17:22:29.920488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:29.920452 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:22:29.972961 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:29.972882 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:30.006905 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.006874 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg5kl\" (UniqueName: \"kubernetes.io/projected/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-kube-api-access-dg5kl\") pod \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\" (UID: \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\") " Jul 10 17:22:30.007107 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.006942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-tls-cert\") pod \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\" (UID: \"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd\") " Jul 10 17:22:30.009072 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.009035 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-kube-api-access-dg5kl" (OuterVolumeSpecName: "kube-api-access-dg5kl") pod "6101b6c6-9caf-41b7-ab2a-8a816d05e0bd" (UID: "6101b6c6-9caf-41b7-ab2a-8a816d05e0bd"). InnerVolumeSpecName "kube-api-access-dg5kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:22:30.018746 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.018697 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6101b6c6-9caf-41b7-ab2a-8a816d05e0bd" (UID: "6101b6c6-9caf-41b7-ab2a-8a816d05e0bd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:22:30.072065 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.072029 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:22:30.108230 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.108200 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg5kl\" (UniqueName: \"kubernetes.io/projected/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-kube-api-access-dg5kl\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:22:30.108230 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.108225 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd-tls-cert\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:22:30.629904 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.629870 2580 generic.go:356] "Generic (PLEG): container finished" podID="6101b6c6-9caf-41b7-ab2a-8a816d05e0bd" containerID="65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4" exitCode=0 Jul 10 17:22:30.630330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.629932 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" Jul 10 17:22:30.630330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.629962 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" event={"ID":"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd","Type":"ContainerDied","Data":"65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4"} Jul 10 17:22:30.630330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.630021 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc58b6dc-5s8lr" event={"ID":"6101b6c6-9caf-41b7-ab2a-8a816d05e0bd","Type":"ContainerDied","Data":"2a09daa6bfaf36c22252cd42269d3668258b34556ba2d558b3963c9db8a6f0f0"} Jul 10 17:22:30.630330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.630047 2580 scope.go:117] "RemoveContainer" containerID="65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4" Jul 10 17:22:30.638896 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.638875 2580 scope.go:117] "RemoveContainer" containerID="65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4" Jul 10 17:22:30.639189 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:22:30.639168 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4\": container with ID starting with 65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4 not found: ID does not exist" containerID="65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4" Jul 10 17:22:30.639256 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.639202 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4"} err="failed to get container status \"65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4\": rpc error: code = NotFound desc = could not find container \"65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4\": container with ID starting with 65d7ec6f667f55fb2cdb3fedccfa5a32391de744cd04759906f2152d9b3e01e4 not found: ID does not exist" Jul 10 17:22:30.657332 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.657293 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bc58b6dc-5s8lr"] Jul 10 17:22:30.662176 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:30.662151 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5bc58b6dc-5s8lr"] Jul 10 17:22:32.501596 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:32.501567 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6101b6c6-9caf-41b7-ab2a-8a816d05e0bd" path="/var/lib/kubelet/pods/6101b6c6-9caf-41b7-ab2a-8a816d05e0bd/volumes" Jul 10 17:22:56.657856 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:22:56.657818 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:23:06.755075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:23:06.754992 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:23:18.053442 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:23:18.053408 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:23:24.881389 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:23:24.881351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:24.982157 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:23:24.982124 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:26.856056 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:23:26.856025 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:23:38.555774 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:23:38.555737 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:23:40.061224 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:23:40.061170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:23:41.546391 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:23:41.546356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:02.961023 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:02.960987 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:24:17.452749 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:17.452696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:24:21.151222 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:21.151187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:22.022063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:22.022031 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:22.119808 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:22.119779 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:22.242535 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:22.242506 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:27.914154 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:27.914121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:27.973850 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:27.973821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:28.062657 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:28.062632 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:28.161941 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:28.161918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:28.262367 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:28.262298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:33.863653 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:33.863618 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:33.926344 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:33.926309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:24:49.555871 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:49.555794 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:24:57.757284 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:24:57.757247 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:25:08.354357 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:25:08.354317 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:25:15.964651 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:25:15.964617 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:25:26.154837 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:25:26.154799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:25:41.754936 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:25:41.754901 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:25:51.752277 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:25:51.752240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:25:59.854367 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:25:59.854331 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:26:10.459002 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:26:10.458912 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:26:12.229094 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:26:12.229060 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:13.344199 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:26:13.344166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:14.456132 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:26:14.456100 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:26:14.462741 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:26:14.462692 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:26:21.560457 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:26:21.560422 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:26:30.553228 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:26:30.553194 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:26:45.050903 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:26:45.050864 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:26:52.947894 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:26:52.947860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:27:03.650235 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:27:03.650202 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:27:20.559302 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:27:20.559270 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:27:29.955699 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:27:29.955658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:27:39.749396 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:27:39.749307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:27:43.068961 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:27:43.068920 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:27:49.249401 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:27:49.249368 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:27:58.746357 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:27:58.746319 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:28:19.508699 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:28:19.508659 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:29:48.953179 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:29:48.953144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:30:00.956487 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:30:00.956454 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:30:10.249998 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:30:10.249961 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:30:21.150443 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:30:21.150406 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:30:32.957618 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:30:32.957583 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:30:42.845822 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:30:42.845785 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:30:52.650433 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:30:52.650399 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:31:02.256925 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:31:02.256888 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:31:13.551685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:31:13.551648 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:31:14.487415 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:31:14.487383 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:31:14.491526 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:31:14.491497 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:31:22.157050 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:31:22.157013 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:31:33.148764 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:31:33.148719 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:31:57.259212 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:31:57.259173 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:32:04.764911 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:32:04.764873 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:32:14.663131 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:32:14.663040 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:32:26.969016 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:32:26.968980 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:32:43.561981 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:32:43.561952 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:32:53.352694 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:32:53.352655 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:33:03.054564 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:33:03.054527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:33:13.448762 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:33:13.448723 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:33:22.251569 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:33:22.251532 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:33:31.951538 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:33:31.951502 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:33:45.346193 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:33:45.346110 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:33:54.251841 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:33:54.251801 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:34:04.452390 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:34:04.452354 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:34:13.853089 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:34:13.853051 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:34:14.433656 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:34:14.433624 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:26.149214 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:34:26.149174 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:34:35.555406 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:34:35.555349 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:34:44.164398 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:34:44.164360 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:47.554528 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:34:47.554477 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:34:55.933661 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:34:55.933620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:34:59.151622 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:34:59.151586 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:35:09.045497 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:09.045405 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:35:11.132228 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:11.132190 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:18.575024 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:18.574995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:21.303447 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:21.303411 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:21.754052 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:21.754014 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:35:26.813481 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:26.813445 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:31.894291 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:31.894258 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:34.631247 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:34.631210 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:34.730800 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:34.730765 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:35.077879 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:35.077793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:36.549440 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:36.549399 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:35:40.851276 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:40.851242 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:40.949064 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:40.949028 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:41.042483 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:41.042438 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:41.105144 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:41.105054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:42.140976 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:42.140935 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:42.242146 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:42.242109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:42.342511 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:42.342459 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:42.628111 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:42.628072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:42.792893 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:42.792853 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:42.901393 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:42.901309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:44.670443 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:44.670417 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:49.755822 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:49.755788 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:35:55.179719 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:55.179673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:35:57.642176 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:35:57.642145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:01.056547 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:01.056504 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:36:02.294273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:02.294237 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:07.932368 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:07.932335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:08.033631 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:08.033598 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:12.501508 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:12.501465 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:12.784105 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:12.784069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:13.989982 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:13.989952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:14.096923 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:14.096888 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:14.253283 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:14.253182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:14.525145 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:14.525061 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:36:14.529092 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:14.529065 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:36:14.603820 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:14.603783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:14.702691 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:14.702656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:14.803515 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:14.803432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:16.653647 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:16.653608 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:36:19.230067 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:19.230040 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:19.331765 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:19.331737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:36:22.689085 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:36:22.689056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:29.736173 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:29.736134 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:32.349871 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:32.349840 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:37:34.209991 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:34.209942 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:39.839028 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:39.838990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:39.898788 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:39.898761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:45.168814 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:45.168777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:45.468721 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:45.468681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:46.573192 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:46.573152 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:46.668865 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:46.668833 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:46.793555 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:46.793521 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:47.179719 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:47.179684 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:47.282560 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:47.282531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:47.382497 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:47.382461 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:49.448893 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:49.448859 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:37:51.781120 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:51.781033 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:51.835618 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:51.835588 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:37:54.673055 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:37:54.673003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:01.467029 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:01.466992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:02.262866 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:02.262830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:02.364296 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:02.364268 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:04.457834 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:04.457804 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:39:06.120045 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:06.120011 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:12.158189 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:12.158150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:12.282646 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:12.282612 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:16.314213 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:16.314176 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:17.777939 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:17.777903 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:18.483891 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:18.483851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:22.618808 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:22.618776 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:24.241232 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:24.241194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:24.297328 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:24.297296 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:24.386674 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:24.386637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:24.485649 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:24.485620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:24.586042 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:24.585970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:24.932980 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:24.932947 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:50.860184 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:50.860114 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:56.554433 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:56.554396 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:56.612859 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:56.612826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:56.693651 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:56.693623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:56.793213 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:39:56.793189 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:12.652051 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:12.652015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:18.365253 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:18.365216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:18.472901 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:18.472868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:18.613060 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:18.613031 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:29.091628 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:29.091588 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:29.873520 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:29.873484 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:35.621882 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:35.621843 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:35.724173 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:35.724141 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:36.255075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:36.255044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:36.357289 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:36.357258 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:36.649681 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:36.649646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:36.749116 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:36.749082 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:36.851384 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:36.851351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:38.394938 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:38.394900 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:40.055245 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:40.055208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:40:44.071717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:44.071667 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:44.172347 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:44.172311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:44.736502 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:44.736469 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:49.147214 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:49.147176 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:40:50.597853 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:50.597810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:50.654273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:50.654241 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:50.831210 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:50.831157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:51.135133 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:51.135104 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:51.320488 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:51.320455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:51.378855 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:51.378830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:53.750436 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:53.750405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:53.826696 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:53.826659 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:53.893920 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:53.893893 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:54.281911 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:54.281870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:54.456043 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:54.456008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:54.557469 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:54.557385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:58.240776 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:58.240726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:58.882222 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:58.882192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:58.985986 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:58.985937 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:59.306535 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:40:59.306457 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:00.750550 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:00.750515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:41:04.936610 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:04.936573 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:05.039782 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:05.039747 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:14.554778 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:14.554754 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:41:14.555872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:14.555852 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:41:15.729755 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:15.729697 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:18.651887 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:18.651853 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:41:20.669051 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:20.669013 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:26.124872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:26.124838 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:26.189485 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:26.189453 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:32.319077 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:32.319041 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:32.687288 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:32.687213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:32.825405 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:32.825363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:32.925466 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:32.925415 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:33.097021 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:33.096919 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:38.717741 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:38.717694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:38.779866 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:38.779834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:38.887186 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:38.887153 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:38.987403 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:38.987336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:39.189815 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:39.189785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:44.528000 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:44.527967 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:44.582603 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:44.582576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:44.701294 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:44.701255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:44.801110 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:44.801035 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:52.101863 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:52.101827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:52.426075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:52.426037 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:52.921497 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:52.921460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:54.953834 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:54.953797 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:55.063506 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:55.063465 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:55.242270 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:55.242184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:55.514673 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:55.514580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:55.667619 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:55.667588 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:55.772298 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:55.772225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:41:59.467274 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:41:59.467242 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:42:01.008575 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:01.008541 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:42:04.649826 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.649791 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth"] Jul 10 17:42:04.650302 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.650168 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6101b6c6-9caf-41b7-ab2a-8a816d05e0bd" containerName="authorino" Jul 10 17:42:04.650302 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.650179 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6101b6c6-9caf-41b7-ab2a-8a816d05e0bd" containerName="authorino" Jul 10 17:42:04.650302 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.650257 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6101b6c6-9caf-41b7-ab2a-8a816d05e0bd" containerName="authorino" Jul 10 17:42:04.653334 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.653317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.657063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.657033 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-039c4f53\"/\"e2e-lifecycle-model-039c4f53-kserve-self-signed-certs\"" Jul 10 17:42:04.657063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.657053 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-039c4f53\"/\"openshift-service-ca.crt\"" Jul 10 17:42:04.657063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.657041 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-039c4f53\"/\"kube-root-ca.crt\"" Jul 10 17:42:04.657289 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.657059 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-039c4f53\"/\"default-dockercfg-n2j2z\"" Jul 10 17:42:04.661652 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.661629 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth"] Jul 10 17:42:04.668950 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.668921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:42:04.772898 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.772871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tmp-dir\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.773075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.772905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tls-certs\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.773075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.772932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qhc\" (UniqueName: \"kubernetes.io/projected/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-kube-api-access-68qhc\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.773075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.772958 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-model-cache\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.773075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.772984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-dshm\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.773075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.773030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-home\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.873601 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.873561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tmp-dir\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.873601 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.873600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tls-certs\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.873826 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.873621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qhc\" (UniqueName: \"kubernetes.io/projected/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-kube-api-access-68qhc\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.873826 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.873642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-model-cache\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.873826 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.873660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-dshm\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.873826 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.873691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-home\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.874033 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.873988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tmp-dir\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.874205 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.874044 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-model-cache\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.874205 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.874076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-home\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.875936 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.875915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-dshm\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.876192 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.876175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tls-certs\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.881177 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.881155 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qhc\" (UniqueName: \"kubernetes.io/projected/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-kube-api-access-68qhc\") pod \"e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:04.962961 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:04.962896 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:05.091881 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:05.091853 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth"] Jul 10 17:42:05.093151 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:42:05.093129 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeb90bbe_0787_44a1_8f98_c73f7d6dc893.slice/crio-ae048c182b554124dee85cdf6440acc1e5ab34dc77f9808671e762db31dfa03c WatchSource:0}: Error finding container ae048c182b554124dee85cdf6440acc1e5ab34dc77f9808671e762db31dfa03c: Status 404 returned error can't find the container with id ae048c182b554124dee85cdf6440acc1e5ab34dc77f9808671e762db31dfa03c Jul 10 17:42:05.157898 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:05.157866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" event={"ID":"eeb90bbe-0787-44a1-8f98-c73f7d6dc893","Type":"ContainerStarted","Data":"8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592"} Jul 10 17:42:05.157999 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:05.157917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" event={"ID":"eeb90bbe-0787-44a1-8f98-c73f7d6dc893","Type":"ContainerStarted","Data":"ae048c182b554124dee85cdf6440acc1e5ab34dc77f9808671e762db31dfa03c"} Jul 10 17:42:05.174011 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:05.173967 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" podStartSLOduration=1.17395241 podStartE2EDuration="1.17395241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:42:05.173350998 +0000 UTC m=+1851.242548099" watchObservedRunningTime="2026-07-10 17:42:05.17395241 +0000 UTC m=+1851.243149510" Jul 10 17:42:09.949432 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:09.949395 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:42:14.963188 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:14.963154 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:14.963648 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:14.963218 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:14.977428 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:14.977405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:15.217049 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:15.216971 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:42:24.357781 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:24.357746 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:42:38.306061 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:38.305932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:42:41.845662 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:41.845629 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:42:42.912821 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:42.912787 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:42:47.892631 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:47.892593 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth"] Jul 10 17:42:47.988510 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:42:47.988478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:02.913287 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:02.913242 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" podUID="eeb90bbe-0787-44a1-8f98-c73f7d6dc893" containerName="main" containerID="cri-o://8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592" gracePeriod=45 Jul 10 17:43:03.168649 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.168593 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:43:03.311191 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311163 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-model-cache\") pod \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " Jul 10 17:43:03.311385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311210 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tmp-dir\") pod \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " Jul 10 17:43:03.311385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311239 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tls-certs\") pod \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " Jul 10 17:43:03.311385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311253 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-home\") pod \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " Jul 10 17:43:03.311385 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311280 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68qhc\" (UniqueName: \"kubernetes.io/projected/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-kube-api-access-68qhc\") pod \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " Jul 10 17:43:03.311641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311452 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-model-cache" (OuterVolumeSpecName: "model-cache") pod "eeb90bbe-0787-44a1-8f98-c73f7d6dc893" (UID: "eeb90bbe-0787-44a1-8f98-c73f7d6dc893"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:43:03.311641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311473 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-dshm\") pod \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\" (UID: \"eeb90bbe-0787-44a1-8f98-c73f7d6dc893\") " Jul 10 17:43:03.311641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311471 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eeb90bbe-0787-44a1-8f98-c73f7d6dc893" (UID: "eeb90bbe-0787-44a1-8f98-c73f7d6dc893"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:43:03.311641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311588 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-home" (OuterVolumeSpecName: "home") pod "eeb90bbe-0787-44a1-8f98-c73f7d6dc893" (UID: "eeb90bbe-0787-44a1-8f98-c73f7d6dc893"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:43:03.311878 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311825 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:43:03.311878 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311842 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:43:03.311878 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.311850 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:43:03.313353 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.313329 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-dshm" (OuterVolumeSpecName: "dshm") pod "eeb90bbe-0787-44a1-8f98-c73f7d6dc893" (UID: "eeb90bbe-0787-44a1-8f98-c73f7d6dc893"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:43:03.313471 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.313450 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-kube-api-access-68qhc" (OuterVolumeSpecName: "kube-api-access-68qhc") pod "eeb90bbe-0787-44a1-8f98-c73f7d6dc893" (UID: "eeb90bbe-0787-44a1-8f98-c73f7d6dc893"). InnerVolumeSpecName "kube-api-access-68qhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:43:03.313516 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.313493 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eeb90bbe-0787-44a1-8f98-c73f7d6dc893" (UID: "eeb90bbe-0787-44a1-8f98-c73f7d6dc893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:43:03.413200 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.413174 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:43:03.413200 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.413197 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qhc\" (UniqueName: \"kubernetes.io/projected/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-kube-api-access-68qhc\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:43:03.413328 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.413206 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb90bbe-0787-44a1-8f98-c73f7d6dc893-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:43:03.436075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.436008 2580 generic.go:356] "Generic (PLEG): container finished" podID="eeb90bbe-0787-44a1-8f98-c73f7d6dc893" containerID="8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592" exitCode=0 Jul 10 17:43:03.436075 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.436053 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" Jul 10 17:43:03.436234 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.436087 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" event={"ID":"eeb90bbe-0787-44a1-8f98-c73f7d6dc893","Type":"ContainerDied","Data":"8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592"} Jul 10 17:43:03.436234 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.436130 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth" event={"ID":"eeb90bbe-0787-44a1-8f98-c73f7d6dc893","Type":"ContainerDied","Data":"ae048c182b554124dee85cdf6440acc1e5ab34dc77f9808671e762db31dfa03c"} Jul 10 17:43:03.436234 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.436147 2580 scope.go:117] "RemoveContainer" containerID="8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592" Jul 10 17:43:03.445350 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.445335 2580 scope.go:117] "RemoveContainer" containerID="8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592" Jul 10 17:43:03.445594 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:43:03.445575 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592\": container with ID starting with 8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592 not found: ID does not exist" containerID="8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592" Jul 10 17:43:03.445653 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.445601 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592"} err="failed to get container status \"8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592\": rpc error: code = NotFound desc = could not find container \"8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592\": container with ID starting with 8b39bd7fa8f1bb8a553ff5174526a36898158524bd5f7b5a340156a350455592 not found: ID does not exist" Jul 10 17:43:03.459867 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.459845 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth"] Jul 10 17:43:03.463759 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:03.463740 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-039c4f53/e2e-lifecycle-model-039c4f53-kserve-d8f74567-xdvth"] Jul 10 17:43:04.502255 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:04.502225 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeb90bbe-0787-44a1-8f98-c73f7d6dc893" path="/var/lib/kubelet/pods/eeb90bbe-0787-44a1-8f98-c73f7d6dc893/volumes" Jul 10 17:43:24.631683 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:24.631652 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:24.992529 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:24.992443 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:30.600864 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:30.600830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:30.701467 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:30.701432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:35.265289 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:35.265255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:35.498197 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:35.498167 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.573637 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:36.573608 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.673582 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:36.673552 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.778160 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:36.778135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:37.595795 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:37.595756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:37.688902 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:37.688871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:37.788050 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:37.788011 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:38.950360 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:38.950325 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:43:39.212080 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:39.211996 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:45.082779 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:45.082736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:45.145768 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:45.145699 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:45.787532 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:45.787501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:51.845940 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:51.845904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:51.962602 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:51.962564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:52.062076 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:52.062042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:53.653717 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:43:53.653667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:44:02.577602 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:02.577568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:02.910872 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:02.910839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:03.011837 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:03.011802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:03.111974 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:03.111942 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:23.866697 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:23.866637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:29.659222 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:29.659189 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:29.719381 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:29.719352 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:30.014441 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:30.014357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:30.377495 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:30.377461 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:30.558482 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:30.558447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:33.689750 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:33.689685 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:33.780193 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:33.780154 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:33.892853 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:33.892821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:34.343847 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:34.343813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:34.444063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:34.444033 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:34.544169 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:34.544131 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:38.185583 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:38.185550 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:38.517363 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:38.517275 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:38.819867 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:38.819790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:38.965668 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:38.965631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:41.552433 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:41.552397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:41.632824 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:41.632793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:41.710397 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:41.710367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:42.066318 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:42.066282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:44:42.069470 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:42.069447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:42.289007 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:42.288974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:42.385557 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:42.385527 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:46.032726 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:46.032669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:55.047564 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.047531 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9"] Jul 10 17:44:55.048096 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.047918 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb90bbe-0787-44a1-8f98-c73f7d6dc893" containerName="main" Jul 10 17:44:55.048096 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.047930 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb90bbe-0787-44a1-8f98-c73f7d6dc893" containerName="main" Jul 10 17:44:55.048096 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.048000 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb90bbe-0787-44a1-8f98-c73f7d6dc893" containerName="main" Jul 10 17:44:55.050893 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.050877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.054433 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.054408 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a3eb74\"/\"kube-root-ca.crt\"" Jul 10 17:44:55.054573 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.054453 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a3eb74\"/\"openshift-service-ca.crt\"" Jul 10 17:44:55.054573 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.054409 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a3eb74\"/\"test-model-a3eb74-kserve-self-signed-certs\"" Jul 10 17:44:55.054573 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.054408 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a3eb74\"/\"default-dockercfg-6k476\"" Jul 10 17:44:55.059811 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.059790 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9"] Jul 10 17:44:55.071611 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.071587 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:44:55.074750 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.074698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:55.107419 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.107394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-dshm\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.107524 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.107424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tmp-dir\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.107524 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.107447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tls-certs\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.107641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.107600 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-kube-api-access-p4vdw\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.107695 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.107670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-model-cache\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.107794 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.107774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-home\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.173752 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.173691 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:55.208838 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.208805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-home\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.208985 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.208861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-dshm\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.208985 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.208882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tmp-dir\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.208985 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.208902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tls-certs\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.209138 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.208985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-kube-api-access-p4vdw\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.209138 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.209029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-model-cache\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.209286 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.209263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-home\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.209354 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.209306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tmp-dir\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.209474 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.209439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-model-cache\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.211169 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.211146 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-dshm\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.211421 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.211403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tls-certs\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.217414 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.217396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-kube-api-access-p4vdw\") pod \"test-model-a3eb74-kserve-5865db9f7-5xnn9\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.274001 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.273970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:55.361338 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.361258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:44:55.492800 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.492778 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9"] Jul 10 17:44:55.494696 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:44:55.494671 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1acad5d0_0a13_41eb_bd0a_3b69463e4f1d.slice/crio-ea3cb265ae6fe8531df736ffdcb1ca4cca759820f4dc338fb6703110062f23b7 WatchSource:0}: Error finding container ea3cb265ae6fe8531df736ffdcb1ca4cca759820f4dc338fb6703110062f23b7: Status 404 returned error can't find the container with id ea3cb265ae6fe8531df736ffdcb1ca4cca759820f4dc338fb6703110062f23b7 Jul 10 17:44:55.951142 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.951105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" event={"ID":"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d","Type":"ContainerStarted","Data":"38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf"} Jul 10 17:44:55.951142 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.951155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" event={"ID":"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d","Type":"ContainerStarted","Data":"ea3cb265ae6fe8531df736ffdcb1ca4cca759820f4dc338fb6703110062f23b7"} Jul 10 17:44:55.968238 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:44:55.968176 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" podStartSLOduration=0.968153316 podStartE2EDuration="968.153316ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:44:55.967549437 +0000 UTC m=+2022.036746533" watchObservedRunningTime="2026-07-10 17:44:55.968153316 +0000 UTC m=+2022.037350416" Jul 10 17:45:00.316330 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:00.316266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:00.378950 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:00.378915 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:03.238369 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:03.238338 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:03.375919 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:03.375882 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:05.361762 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:05.361727 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:45:05.362179 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:05.361778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:45:05.374146 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:05.374123 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:45:06.012772 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:06.012741 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:45:11.044634 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:11.044604 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:45:18.152760 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.152730 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw"] Jul 10 17:45:18.156141 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.156120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.159993 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.159970 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-050e5c\"/\"default-dockercfg-gmdq7\"" Jul 10 17:45:18.160126 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.159970 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-050e5c\"/\"test-model-050e5c-kserve-self-signed-certs\"" Jul 10 17:45:18.160126 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.160014 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-050e5c\"/\"openshift-service-ca.crt\"" Jul 10 17:45:18.160126 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.159971 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-050e5c\"/\"kube-root-ca.crt\"" Jul 10 17:45:18.164373 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.164353 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw"] Jul 10 17:45:18.181341 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.181318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:18.318327 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.318305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-dshm\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.318452 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.318332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tls-certs\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.318452 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.318374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-home\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.318452 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.318423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bc8x\" (UniqueName: \"kubernetes.io/projected/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-kube-api-access-5bc8x\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.318615 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.318516 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tmp-dir\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.318615 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.318573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-model-cache\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.419906 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.419848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tmp-dir\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.419906 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.419893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-model-cache\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.420064 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.419909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-dshm\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.420064 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.419923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tls-certs\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.420064 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.419961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-home\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.420064 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.419976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bc8x\" (UniqueName: \"kubernetes.io/projected/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-kube-api-access-5bc8x\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.420323 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.420301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tmp-dir\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.420393 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.420361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-model-cache\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.420435 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.420410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-home\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.422200 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.422183 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-dshm\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.422388 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.422372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tls-certs\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.429482 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.429463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bc8x\" (UniqueName: \"kubernetes.io/projected/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-kube-api-access-5bc8x\") pod \"test-model-050e5c-kserve-5b857f654f-8hdqw\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.466519 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.466497 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:18.598126 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:18.598099 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw"] Jul 10 17:45:18.599922 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:45:18.599888 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa24ccdb_9f0d_40f7_99dc_6337b535a9b7.slice/crio-4a37ca5d77042146246ed767423e146db7da8cde21bb6146b5f2fb6681b5658b WatchSource:0}: Error finding container 4a37ca5d77042146246ed767423e146db7da8cde21bb6146b5f2fb6681b5658b: Status 404 returned error can't find the container with id 4a37ca5d77042146246ed767423e146db7da8cde21bb6146b5f2fb6681b5658b Jul 10 17:45:19.061726 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:19.061677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" event={"ID":"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7","Type":"ContainerStarted","Data":"b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7"} Jul 10 17:45:19.061882 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:19.061737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" event={"ID":"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7","Type":"ContainerStarted","Data":"4a37ca5d77042146246ed767423e146db7da8cde21bb6146b5f2fb6681b5658b"} Jul 10 17:45:19.078442 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:19.078398 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" podStartSLOduration=1.078383304 podStartE2EDuration="1.078383304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:45:19.076328465 +0000 UTC m=+2045.145525565" watchObservedRunningTime="2026-07-10 17:45:19.078383304 +0000 UTC m=+2045.147580404" Jul 10 17:45:23.943684 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:23.943646 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:45:28.466737 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:28.466688 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:28.467215 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:28.466764 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:28.479208 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:28.479189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:29.123716 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:29.123671 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:45:38.258156 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:38.258070 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:45:49.414145 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:49.411813 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9"] Jul 10 17:45:49.420673 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:49.420645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:50.467162 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:50.467129 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw"] Jul 10 17:45:50.480585 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:50.480531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:51.186326 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:51.186297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:51.288003 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:51.287948 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:56.248986 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:45:56.248955 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:46:01.025100 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:01.025068 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:04.434478 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.434418 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" podUID="1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" containerName="main" containerID="cri-o://38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf" gracePeriod=45 Jul 10 17:46:04.683430 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.683404 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:46:04.844976 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.844914 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-kube-api-access-p4vdw\") pod \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " Jul 10 17:46:04.844976 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.844960 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-model-cache\") pod \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " Jul 10 17:46:04.845170 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.844991 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tls-certs\") pod \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " Jul 10 17:46:04.845170 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.845037 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tmp-dir\") pod \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " Jul 10 17:46:04.845170 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.845068 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-dshm\") pod \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " Jul 10 17:46:04.845170 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.845097 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-home\") pod \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\" (UID: \"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d\") " Jul 10 17:46:04.845390 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.845262 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" (UID: "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:04.845390 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.845313 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" (UID: "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:04.845390 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.845382 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:04.845540 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.845449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-home" (OuterVolumeSpecName: "home") pod "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" (UID: "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:04.847149 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.847126 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-dshm" (OuterVolumeSpecName: "dshm") pod "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" (UID: "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:04.847269 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.847239 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" (UID: "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:46:04.847327 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.847276 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-kube-api-access-p4vdw" (OuterVolumeSpecName: "kube-api-access-p4vdw") pod "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" (UID: "1acad5d0-0a13-41eb-bd0a-3b69463e4f1d"). InnerVolumeSpecName "kube-api-access-p4vdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:46:04.945890 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.945867 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:04.945890 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.945887 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:04.945890 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.945895 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:04.946068 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.945904 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4vdw\" (UniqueName: \"kubernetes.io/projected/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-kube-api-access-p4vdw\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:04.946068 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:04.945912 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:05.287522 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.287489 2580 generic.go:356] "Generic (PLEG): container finished" podID="1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" containerID="38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf" exitCode=0 Jul 10 17:46:05.287743 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.287545 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" Jul 10 17:46:05.287743 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.287573 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" event={"ID":"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d","Type":"ContainerDied","Data":"38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf"} Jul 10 17:46:05.287743 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.287616 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9" event={"ID":"1acad5d0-0a13-41eb-bd0a-3b69463e4f1d","Type":"ContainerDied","Data":"ea3cb265ae6fe8531df736ffdcb1ca4cca759820f4dc338fb6703110062f23b7"} Jul 10 17:46:05.287743 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.287632 2580 scope.go:117] "RemoveContainer" containerID="38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf" Jul 10 17:46:05.298630 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.298610 2580 scope.go:117] "RemoveContainer" containerID="38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf" Jul 10 17:46:05.298994 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:46:05.298966 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf\": container with ID starting with 38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf not found: ID does not exist" containerID="38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf" Jul 10 17:46:05.299064 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.299005 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf"} err="failed to get container status \"38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf\": rpc error: code = NotFound desc = could not find container \"38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf\": container with ID starting with 38eb9c847b4fea8bd1cdff51a24c24dff1c1cf47497fd5c6079cd9d65f5683cf not found: ID does not exist" Jul 10 17:46:05.307578 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.307554 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9"] Jul 10 17:46:05.313403 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.313381 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a3eb74/test-model-a3eb74-kserve-5865db9f7-5xnn9"] Jul 10 17:46:05.489908 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.489877 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" podUID="aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" containerName="main" containerID="cri-o://b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7" gracePeriod=45 Jul 10 17:46:05.676434 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.676404 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:05.739129 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.739108 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:46:05.855497 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.855465 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-home\") pod \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " Jul 10 17:46:05.855670 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.855511 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-model-cache\") pod \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " Jul 10 17:46:05.855670 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.855621 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tmp-dir\") pod \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " Jul 10 17:46:05.855670 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.855648 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-dshm\") pod \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " Jul 10 17:46:05.855848 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.855680 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bc8x\" (UniqueName: \"kubernetes.io/projected/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-kube-api-access-5bc8x\") pod \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " Jul 10 17:46:05.855848 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.855757 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tls-certs\") pod \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\" (UID: \"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7\") " Jul 10 17:46:05.855848 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.855806 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-home" (OuterVolumeSpecName: "home") pod "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" (UID: "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:05.856002 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.855859 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" (UID: "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:05.856002 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.855878 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" (UID: "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:05.856119 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.856102 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:05.856152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.856121 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:05.856152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.856129 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:05.858028 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.857976 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-dshm" (OuterVolumeSpecName: "dshm") pod "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" (UID: "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:05.858097 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.858045 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-kube-api-access-5bc8x" (OuterVolumeSpecName: "kube-api-access-5bc8x") pod "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" (UID: "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7"). InnerVolumeSpecName "kube-api-access-5bc8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:46:05.858097 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.858049 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" (UID: "aa24ccdb-9f0d-40f7-99dc-6337b535a9b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:46:05.957038 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.957013 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:05.957038 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.957036 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bc8x\" (UniqueName: \"kubernetes.io/projected/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-kube-api-access-5bc8x\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:05.957178 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:05.957046 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:46:06.293313 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.293284 2580 generic.go:356] "Generic (PLEG): container finished" podID="aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" containerID="b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7" exitCode=0 Jul 10 17:46:06.293481 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.293333 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" Jul 10 17:46:06.293481 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.293334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" event={"ID":"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7","Type":"ContainerDied","Data":"b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7"} Jul 10 17:46:06.293481 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.293388 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw" event={"ID":"aa24ccdb-9f0d-40f7-99dc-6337b535a9b7","Type":"ContainerDied","Data":"4a37ca5d77042146246ed767423e146db7da8cde21bb6146b5f2fb6681b5658b"} Jul 10 17:46:06.293481 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.293413 2580 scope.go:117] "RemoveContainer" containerID="b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7" Jul 10 17:46:06.306076 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.306057 2580 scope.go:117] "RemoveContainer" containerID="b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7" Jul 10 17:46:06.306348 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:46:06.306324 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7\": container with ID starting with b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7 not found: ID does not exist" containerID="b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7" Jul 10 17:46:06.306394 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.306358 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7"} err="failed to get container status \"b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7\": rpc error: code = NotFound desc = could not find container \"b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7\": container with ID starting with b4756e8f0e016244e031b0d4cc5cd05b13ce3ca552917d00ed1877dab23634f7 not found: ID does not exist" Jul 10 17:46:06.317867 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.317847 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw"] Jul 10 17:46:06.322124 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.322105 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-050e5c/test-model-050e5c-kserve-5b857f654f-8hdqw"] Jul 10 17:46:06.502532 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.502500 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" path="/var/lib/kubelet/pods/1acad5d0-0a13-41eb-bd0a-3b69463e4f1d/volumes" Jul 10 17:46:06.502996 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:06.502979 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" path="/var/lib/kubelet/pods/aa24ccdb-9f0d-40f7-99dc-6337b535a9b7/volumes" Jul 10 17:46:07.546212 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:07.546177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:46:11.380977 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:11.380942 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:11.435384 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:11.435358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:14.585175 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:14.585144 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:46:14.586378 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:14.586353 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:46:16.390005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:16.389974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:16.613005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:16.612968 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:17.369102 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:17.369070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:17.469987 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:17.469954 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:27.143462 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:27.143431 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:30.451289 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:30.451255 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:46:31.815030 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:31.814989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:37.857405 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:37.857350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:37.918807 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:37.918781 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:42.521542 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:42.521511 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:42.735810 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:42.735780 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:43.127126 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:43.127081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:43.551365 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:43.551281 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:43.638242 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:43.638203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:46.409887 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:46.409849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:46.504766 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:46.504700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:46.622945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:46.622916 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:46.767997 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:46.767897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:47.119898 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:47.119817 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:47.220742 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:47.220692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:50.260918 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:50.260885 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:51.174820 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:51.174742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:51.462100 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:51.462005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:51.654274 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:51.654240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:46:51.771723 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:51.771634 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:51.865602 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:51.865569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:54.387528 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:54.387487 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:54.484878 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:54.484849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:54.566566 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:54.566533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:54.720256 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:54.720173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:55.099983 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:55.099895 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:55.201795 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:55.201756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:58.937937 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:46:58.937904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:02.063193 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:02.063159 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:47:07.609607 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:07.609524 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:07.710230 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:07.710195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:15.829671 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:15.829636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:15.931921 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:15.931891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:17.589946 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:17.589911 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:17.689754 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:17.689693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:17.854879 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:17.854792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:22.142349 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:22.142312 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:22.999372 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:22.999338 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:47:28.091217 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:28.091175 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:28.192138 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:28.192110 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:32.813334 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:32.813296 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:33.082334 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:33.082245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:33.742665 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:33.742619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:33.843869 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:33.843833 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:43.838183 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:43.838149 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:46.952659 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:46.952625 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:47:48.295022 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:48.294988 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:54.404508 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:54.404469 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:54.503855 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:54.503823 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:59.139391 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:59.139359 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:59.349511 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:59.349480 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:59.691025 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:47:59.690992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:00.053791 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:00.053667 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:00.410411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:00.410376 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:02.998381 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:02.998348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:03.202188 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:03.202149 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:03.469181 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:03.469150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:03.608470 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:03.608442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:03.973062 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:03.972979 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:04.139745 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:04.139700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:07.449531 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:07.449503 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:48:07.500588 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:07.500549 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:07.867492 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:07.867382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:08.202382 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:08.202323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:08.548517 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:08.548415 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:10.999268 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:10.999233 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:11.169499 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:11.169463 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:11.380829 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:11.380800 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:11.586153 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:11.586120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:11.983935 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:11.983906 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:12.104474 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:12.104442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:15.653678 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:15.653611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:24.454518 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:24.454481 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:48:25.955181 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:25.955139 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:26.055636 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:26.055605 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:30.156577 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.156539 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2"] Jul 10 17:48:30.157140 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.157115 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" containerName="main" Jul 10 17:48:30.157140 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.157136 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" containerName="main" Jul 10 17:48:30.157337 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.157168 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" containerName="main" Jul 10 17:48:30.157337 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.157177 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" containerName="main" Jul 10 17:48:30.157337 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.157258 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa24ccdb-9f0d-40f7-99dc-6337b535a9b7" containerName="main" Jul 10 17:48:30.157337 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.157270 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1acad5d0-0a13-41eb-bd0a-3b69463e4f1d" containerName="main" Jul 10 17:48:30.160550 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.160532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.164201 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.164180 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3beb3b\"/\"kube-root-ca.crt\"" Jul 10 17:48:30.164201 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.164204 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3beb3b\"/\"default-dockercfg-ks4mq\"" Jul 10 17:48:30.164401 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.164218 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3beb3b\"/\"openshift-service-ca.crt\"" Jul 10 17:48:30.164401 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.164325 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3beb3b\"/\"auth-test-model-3beb3b-kserve-self-signed-certs\"" Jul 10 17:48:30.172442 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.172418 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2"] Jul 10 17:48:30.192265 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.192232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:30.269152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.269120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-model-cache\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.269285 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.269171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fce773-5830-461a-8a33-2ba67e106280-tls-certs\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.269285 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.269236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-tmp-dir\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.269285 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.269278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-dshm\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.269429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.269300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rr5d\" (UniqueName: \"kubernetes.io/projected/f0fce773-5830-461a-8a33-2ba67e106280-kube-api-access-6rr5d\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.269486 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.269427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-home\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.370088 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.370049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-model-cache\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.370254 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.370127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fce773-5830-461a-8a33-2ba67e106280-tls-certs\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.370254 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.370167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-tmp-dir\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.370254 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.370201 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-dshm\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.370254 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.370228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rr5d\" (UniqueName: \"kubernetes.io/projected/f0fce773-5830-461a-8a33-2ba67e106280-kube-api-access-6rr5d\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.370480 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.370302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-home\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.370537 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.370488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-model-cache\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.370537 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.370503 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-tmp-dir\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.370635 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.370625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-home\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.372333 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.372303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-dshm\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.372694 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.372673 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fce773-5830-461a-8a33-2ba67e106280-tls-certs\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.380192 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.380163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rr5d\" (UniqueName: \"kubernetes.io/projected/f0fce773-5830-461a-8a33-2ba67e106280-kube-api-access-6rr5d\") pod \"auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.470441 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.470361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:30.604835 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.604811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2"] Jul 10 17:48:30.606360 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:48:30.606316 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0fce773_5830_461a_8a33_2ba67e106280.slice/crio-34b080e82662dc2cfdae2922e81fdc3bd0b72970e357af362df11f99bd9bdda1 WatchSource:0}: Error finding container 34b080e82662dc2cfdae2922e81fdc3bd0b72970e357af362df11f99bd9bdda1: Status 404 returned error can't find the container with id 34b080e82662dc2cfdae2922e81fdc3bd0b72970e357af362df11f99bd9bdda1 Jul 10 17:48:30.977573 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.977536 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" event={"ID":"f0fce773-5830-461a-8a33-2ba67e106280","Type":"ContainerStarted","Data":"6eb881c185885c3bf5c99e7cc3a56fc2d16a75e689a3d235bb0d10ce9cc935ef"} Jul 10 17:48:30.977764 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.977580 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" event={"ID":"f0fce773-5830-461a-8a33-2ba67e106280","Type":"ContainerStarted","Data":"34b080e82662dc2cfdae2922e81fdc3bd0b72970e357af362df11f99bd9bdda1"} Jul 10 17:48:30.995203 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:30.995143 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" podStartSLOduration=0.995127723 podStartE2EDuration="995.127723ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:30.993492952 +0000 UTC m=+2237.062690053" watchObservedRunningTime="2026-07-10 17:48:30.995127723 +0000 UTC m=+2237.064324854" Jul 10 17:48:32.869586 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:32.869548 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:32.969721 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:32.969672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:33.853240 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:33.853211 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:48:40.471386 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:40.471293 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:40.471386 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:40.471349 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:40.483379 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:40.483358 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:41.037097 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:41.037064 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:48:49.049790 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.049754 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4"] Jul 10 17:48:49.053589 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.053566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.056312 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.056289 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"kube-root-ca.crt\"" Jul 10 17:48:49.057243 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.057206 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"auth-test-model-9f66f0-kserve-self-signed-certs\"" Jul 10 17:48:49.057353 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.057253 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"default-dockercfg-7td5b\"" Jul 10 17:48:49.057353 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.057309 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"openshift-service-ca.crt\"" Jul 10 17:48:49.063073 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.063051 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4"] Jul 10 17:48:49.076120 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.076085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:49.249566 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.249520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-home\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.249566 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.249560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-model-cache\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.249859 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.249585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-dshm\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.249859 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.249692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tmp-dir\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.249859 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.249781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c72q\" (UniqueName: \"kubernetes.io/projected/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-kube-api-access-5c72q\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.249859 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.249816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tls-certs\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.350866 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.350762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c72q\" (UniqueName: \"kubernetes.io/projected/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-kube-api-access-5c72q\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.350866 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.350820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tls-certs\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.350866 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.350861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-home\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.351163 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.350877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-model-cache\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.351163 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.350895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-dshm\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.351163 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.350958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tmp-dir\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.351291 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.351263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-home\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.351329 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.351289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-model-cache\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.351366 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.351358 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tmp-dir\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.353265 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.353242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-dshm\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.353413 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.353395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tls-certs\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.360510 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.360493 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c72q\" (UniqueName: \"kubernetes.io/projected/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-kube-api-access-5c72q\") pod \"auth-test-model-9f66f0-kserve-78db66b975-788c4\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.364372 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.364354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:49.501079 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:49.501056 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4"] Jul 10 17:48:49.503297 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:48:49.503268 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e8ae76f_9ed5_4f8b_a853_f24b21424cf7.slice/crio-5e8a6974dc21b70774e80686ac8c24b3f7fd28492795b20b14a4042807b98716 WatchSource:0}: Error finding container 5e8a6974dc21b70774e80686ac8c24b3f7fd28492795b20b14a4042807b98716: Status 404 returned error can't find the container with id 5e8a6974dc21b70774e80686ac8c24b3f7fd28492795b20b14a4042807b98716 Jul 10 17:48:50.071531 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:50.071488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" event={"ID":"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7","Type":"ContainerStarted","Data":"213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03"} Jul 10 17:48:50.071939 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:50.071540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" event={"ID":"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7","Type":"ContainerStarted","Data":"5e8a6974dc21b70774e80686ac8c24b3f7fd28492795b20b14a4042807b98716"} Jul 10 17:48:50.091092 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:50.091049 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" podStartSLOduration=1.091034333 podStartE2EDuration="1.091034333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:50.088954156 +0000 UTC m=+2256.158151256" watchObservedRunningTime="2026-07-10 17:48:50.091034333 +0000 UTC m=+2256.160231433" Jul 10 17:48:53.146622 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:53.146577 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:48:59.364670 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:59.364633 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:59.365087 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:59.364686 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:48:59.376986 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:48:59.376954 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:49:00.130856 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:49:00.130825 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:49:03.936169 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:49:03.936136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:04.039935 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:49:04.039904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:20.254734 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:49:20.254687 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:49:36.754560 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:49:36.754527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:49:54.592332 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:49:54.592292 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:50:12.056042 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:50:12.055956 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:50:26.580584 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:50:26.580551 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:27.707227 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:50:27.707192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:29.046227 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:50:29.046192 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:50:43.258392 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:50:43.258353 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:51:01.346852 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:01.346816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:51:06.568132 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:06.568096 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2"] Jul 10 17:51:06.586467 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:06.586428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:07.528394 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:07.528361 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4"] Jul 10 17:51:07.545694 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:07.545662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:13.857952 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.857914 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj"] Jul 10 17:51:13.861512 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.861494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:13.863873 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.863852 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3beb3b\"/\"sub-test-model-3beb3b-kserve-self-signed-certs\"" Jul 10 17:51:13.870091 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.870068 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj"] Jul 10 17:51:13.877023 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.877003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:13.989961 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.989923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwzx\" (UniqueName: \"kubernetes.io/projected/4e08686f-1b42-421f-baad-74610cede365-kube-api-access-qtwzx\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:13.990152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.990003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-home\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:13.990152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.990026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e08686f-1b42-421f-baad-74610cede365-tls-certs\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:13.990152 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.990069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-tmp-dir\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:13.990320 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.990165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-dshm\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:13.990320 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:13.990227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-model-cache\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.091299 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.091269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-home\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.091299 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.091302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e08686f-1b42-421f-baad-74610cede365-tls-certs\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.091542 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.091352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-tmp-dir\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.091542 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.091411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-dshm\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.091542 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.091456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-model-cache\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.091542 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.091482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwzx\" (UniqueName: \"kubernetes.io/projected/4e08686f-1b42-421f-baad-74610cede365-kube-api-access-qtwzx\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.091786 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.091764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-home\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.091849 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.091825 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-tmp-dir\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.091892 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.091855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-model-cache\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.093611 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.093586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-dshm\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.093971 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.093956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e08686f-1b42-421f-baad-74610cede365-tls-certs\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.098407 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.098384 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwzx\" (UniqueName: \"kubernetes.io/projected/4e08686f-1b42-421f-baad-74610cede365-kube-api-access-qtwzx\") pod \"sub-test-model-3beb3b-kserve-988dff94f-qpdwj\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.171500 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.171466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:14.305000 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.304977 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj"] Jul 10 17:51:14.307473 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:51:14.307445 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e08686f_1b42_421f_baad_74610cede365.slice/crio-1833018778d100af5dff19cd388e107018c605c59f192343ba052e5b50198daa WatchSource:0}: Error finding container 1833018778d100af5dff19cd388e107018c605c59f192343ba052e5b50198daa: Status 404 returned error can't find the container with id 1833018778d100af5dff19cd388e107018c605c59f192343ba052e5b50198daa Jul 10 17:51:14.616127 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.616044 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:51:14.618435 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.618410 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:51:14.762378 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.762345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" event={"ID":"4e08686f-1b42-421f-baad-74610cede365","Type":"ContainerStarted","Data":"4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8"} Jul 10 17:51:14.762378 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.762385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" event={"ID":"4e08686f-1b42-421f-baad-74610cede365","Type":"ContainerStarted","Data":"1833018778d100af5dff19cd388e107018c605c59f192343ba052e5b50198daa"} Jul 10 17:51:14.778207 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:14.778155 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" podStartSLOduration=1.778141514 podStartE2EDuration="1.778141514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:51:14.777471476 +0000 UTC m=+2400.846668577" watchObservedRunningTime="2026-07-10 17:51:14.778141514 +0000 UTC m=+2400.847338616" Jul 10 17:51:20.657790 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:20.657752 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:51:21.588296 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.588241 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" podUID="f0fce773-5830-461a-8a33-2ba67e106280" containerName="main" containerID="cri-o://6eb881c185885c3bf5c99e7cc3a56fc2d16a75e689a3d235bb0d10ce9cc935ef" gracePeriod=45 Jul 10 17:51:21.797791 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.797756 2580 generic.go:356] "Generic (PLEG): container finished" podID="f0fce773-5830-461a-8a33-2ba67e106280" containerID="6eb881c185885c3bf5c99e7cc3a56fc2d16a75e689a3d235bb0d10ce9cc935ef" exitCode=0 Jul 10 17:51:21.798146 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.797836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" event={"ID":"f0fce773-5830-461a-8a33-2ba67e106280","Type":"ContainerDied","Data":"6eb881c185885c3bf5c99e7cc3a56fc2d16a75e689a3d235bb0d10ce9cc935ef"} Jul 10 17:51:21.839945 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.839883 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:51:21.975913 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.975882 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fce773-5830-461a-8a33-2ba67e106280-tls-certs\") pod \"f0fce773-5830-461a-8a33-2ba67e106280\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " Jul 10 17:51:21.976073 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.975922 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-home\") pod \"f0fce773-5830-461a-8a33-2ba67e106280\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " Jul 10 17:51:21.976073 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.975951 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rr5d\" (UniqueName: \"kubernetes.io/projected/f0fce773-5830-461a-8a33-2ba67e106280-kube-api-access-6rr5d\") pod \"f0fce773-5830-461a-8a33-2ba67e106280\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " Jul 10 17:51:21.976073 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.975975 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-model-cache\") pod \"f0fce773-5830-461a-8a33-2ba67e106280\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " Jul 10 17:51:21.976073 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.976011 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-tmp-dir\") pod \"f0fce773-5830-461a-8a33-2ba67e106280\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " Jul 10 17:51:21.976073 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.976045 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-dshm\") pod \"f0fce773-5830-461a-8a33-2ba67e106280\" (UID: \"f0fce773-5830-461a-8a33-2ba67e106280\") " Jul 10 17:51:21.976353 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.976149 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-home" (OuterVolumeSpecName: "home") pod "f0fce773-5830-461a-8a33-2ba67e106280" (UID: "f0fce773-5830-461a-8a33-2ba67e106280"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:51:21.976353 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.976253 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0fce773-5830-461a-8a33-2ba67e106280" (UID: "f0fce773-5830-461a-8a33-2ba67e106280"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:51:21.976353 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.976272 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0fce773-5830-461a-8a33-2ba67e106280" (UID: "f0fce773-5830-461a-8a33-2ba67e106280"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:51:21.976500 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.976432 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:21.976500 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.976453 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:21.976500 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.976468 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:21.978204 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.978173 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0fce773-5830-461a-8a33-2ba67e106280-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0fce773-5830-461a-8a33-2ba67e106280" (UID: "f0fce773-5830-461a-8a33-2ba67e106280"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:51:21.978323 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.978219 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-dshm" (OuterVolumeSpecName: "dshm") pod "f0fce773-5830-461a-8a33-2ba67e106280" (UID: "f0fce773-5830-461a-8a33-2ba67e106280"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:51:21.978323 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:21.978234 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fce773-5830-461a-8a33-2ba67e106280-kube-api-access-6rr5d" (OuterVolumeSpecName: "kube-api-access-6rr5d") pod "f0fce773-5830-461a-8a33-2ba67e106280" (UID: "f0fce773-5830-461a-8a33-2ba67e106280"). InnerVolumeSpecName "kube-api-access-6rr5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:51:22.077895 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.077852 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fce773-5830-461a-8a33-2ba67e106280-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:22.077895 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.077890 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fce773-5830-461a-8a33-2ba67e106280-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:22.077895 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.077900 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rr5d\" (UniqueName: \"kubernetes.io/projected/f0fce773-5830-461a-8a33-2ba67e106280-kube-api-access-6rr5d\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:22.549214 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.549177 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" podUID="5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" containerName="main" containerID="cri-o://213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03" gracePeriod=45 Jul 10 17:51:22.794432 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.794408 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:51:22.802825 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.802796 2580 generic.go:356] "Generic (PLEG): container finished" podID="5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" containerID="213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03" exitCode=0 Jul 10 17:51:22.803166 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.802829 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" Jul 10 17:51:22.803166 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.802876 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" event={"ID":"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7","Type":"ContainerDied","Data":"213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03"} Jul 10 17:51:22.803166 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.802912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4" event={"ID":"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7","Type":"ContainerDied","Data":"5e8a6974dc21b70774e80686ac8c24b3f7fd28492795b20b14a4042807b98716"} Jul 10 17:51:22.803166 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.802925 2580 scope.go:117] "RemoveContainer" containerID="213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03" Jul 10 17:51:22.804098 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.804081 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" Jul 10 17:51:22.804224 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.804132 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2" event={"ID":"f0fce773-5830-461a-8a33-2ba67e106280","Type":"ContainerDied","Data":"34b080e82662dc2cfdae2922e81fdc3bd0b72970e357af362df11f99bd9bdda1"} Jul 10 17:51:22.811460 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.811440 2580 scope.go:117] "RemoveContainer" containerID="213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03" Jul 10 17:51:22.811798 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:51:22.811768 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03\": container with ID starting with 213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03 not found: ID does not exist" containerID="213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03" Jul 10 17:51:22.811897 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.811798 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03"} err="failed to get container status \"213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03\": rpc error: code = NotFound desc = could not find container \"213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03\": container with ID starting with 213f618d7dc79b8da98aa8638a7d564f1897a02142fdd3a981fb0239cb86dd03 not found: ID does not exist" Jul 10 17:51:22.811897 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.811818 2580 scope.go:117] "RemoveContainer" containerID="6eb881c185885c3bf5c99e7cc3a56fc2d16a75e689a3d235bb0d10ce9cc935ef" Jul 10 17:51:22.825294 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.825271 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2"] Jul 10 17:51:22.826941 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.826923 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/auth-test-model-3beb3b-kserve-65878c6b8c-2pkx2"] Jul 10 17:51:22.889346 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.886135 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-dshm\") pod \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " Jul 10 17:51:22.889346 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.886204 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tls-certs\") pod \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " Jul 10 17:51:22.889346 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.886249 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tmp-dir\") pod \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " Jul 10 17:51:22.889346 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.886286 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-model-cache\") pod \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " Jul 10 17:51:22.889346 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.886388 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c72q\" (UniqueName: \"kubernetes.io/projected/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-kube-api-access-5c72q\") pod \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " Jul 10 17:51:22.889346 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.886416 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-home\") pod \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\" (UID: \"5e8ae76f-9ed5-4f8b-a853-f24b21424cf7\") " Jul 10 17:51:22.889346 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.887122 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-home" (OuterVolumeSpecName: "home") pod "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" (UID: "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:51:22.890107 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.890077 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-dshm" (OuterVolumeSpecName: "dshm") pod "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" (UID: "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:51:22.890231 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.890208 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" (UID: "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:51:22.890422 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.890395 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" (UID: "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:51:22.892429 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.892408 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-kube-api-access-5c72q" (OuterVolumeSpecName: "kube-api-access-5c72q") pod "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" (UID: "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7"). InnerVolumeSpecName "kube-api-access-5c72q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:51:22.894014 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.893992 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" (UID: "5e8ae76f-9ed5-4f8b-a853-f24b21424cf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:51:22.987557 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.987534 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:22.987557 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.987558 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:22.987686 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.987568 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:22.987686 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.987576 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:22.987686 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.987585 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c72q\" (UniqueName: \"kubernetes.io/projected/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-kube-api-access-5c72q\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:22.987686 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:22.987592 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:51:23.124929 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:23.124900 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4"] Jul 10 17:51:23.128761 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:23.128732 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/auth-test-model-9f66f0-kserve-78db66b975-788c4"] Jul 10 17:51:24.171997 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:24.171949 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:24.172428 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:24.172018 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:24.184577 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:24.184552 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:24.503107 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:24.503026 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" path="/var/lib/kubelet/pods/5e8ae76f-9ed5-4f8b-a853-f24b21424cf7/volumes" Jul 10 17:51:24.503415 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:24.503402 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0fce773-5830-461a-8a33-2ba67e106280" path="/var/lib/kubelet/pods/f0fce773-5830-461a-8a33-2ba67e106280/volumes" Jul 10 17:51:24.828421 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:24.828335 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:51:39.851990 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.851904 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw"] Jul 10 17:51:39.854273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.852490 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" containerName="main" Jul 10 17:51:39.854273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.852510 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" containerName="main" Jul 10 17:51:39.854273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.852543 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0fce773-5830-461a-8a33-2ba67e106280" containerName="main" Jul 10 17:51:39.854273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.852552 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0fce773-5830-461a-8a33-2ba67e106280" containerName="main" Jul 10 17:51:39.854273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.852664 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8ae76f-9ed5-4f8b-a853-f24b21424cf7" containerName="main" Jul 10 17:51:39.854273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.852682 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0fce773-5830-461a-8a33-2ba67e106280" containerName="main" Jul 10 17:51:39.856030 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.856011 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:39.859941 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.859914 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"openshift-service-ca.crt\"" Jul 10 17:51:39.860101 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.859914 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"sub-test-model-9f66f0-kserve-self-signed-certs\"" Jul 10 17:51:39.860101 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.859970 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"kube-root-ca.crt\"" Jul 10 17:51:39.860101 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.860004 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"default-dockercfg-7td5b\"" Jul 10 17:51:39.865364 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.865343 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw"] Jul 10 17:51:39.889968 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.889924 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:39.946972 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.946943 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-dshm\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:39.947157 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.947003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-home\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:39.947157 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.947049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-tmp-dir\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:39.947157 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.947071 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-model-cache\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:39.947157 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.947085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dca30e-711d-43e1-99c6-32c146cc324a-tls-certs\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:39.947349 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:39.947148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zs5f\" (UniqueName: \"kubernetes.io/projected/a5dca30e-711d-43e1-99c6-32c146cc324a-kube-api-access-9zs5f\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.048258 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.048214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zs5f\" (UniqueName: \"kubernetes.io/projected/a5dca30e-711d-43e1-99c6-32c146cc324a-kube-api-access-9zs5f\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.048444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.048310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-dshm\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.048444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.048368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-home\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.048444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.048410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-tmp-dir\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.048444 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.048440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-model-cache\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.048641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.048555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dca30e-711d-43e1-99c6-32c146cc324a-tls-certs\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.048879 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.048856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-model-cache\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.048970 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.048879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-home\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.048970 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.048921 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-tmp-dir\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.050634 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.050612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-dshm\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.050825 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.050810 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dca30e-711d-43e1-99c6-32c146cc324a-tls-certs\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.055377 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.055361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zs5f\" (UniqueName: \"kubernetes.io/projected/a5dca30e-711d-43e1-99c6-32c146cc324a-kube-api-access-9zs5f\") pod \"sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.167371 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.167338 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:40.328817 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.328784 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw"] Jul 10 17:51:40.330823 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:51:40.330787 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5dca30e_711d_43e1_99c6_32c146cc324a.slice/crio-3248ec05b074f1a6c41266f205ab8c6728a148f5f1e98b733e45449ad9f804c1 WatchSource:0}: Error finding container 3248ec05b074f1a6c41266f205ab8c6728a148f5f1e98b733e45449ad9f804c1: Status 404 returned error can't find the container with id 3248ec05b074f1a6c41266f205ab8c6728a148f5f1e98b733e45449ad9f804c1 Jul 10 17:51:40.892174 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.892142 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" event={"ID":"a5dca30e-711d-43e1-99c6-32c146cc324a","Type":"ContainerStarted","Data":"ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8"} Jul 10 17:51:40.892174 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.892182 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" event={"ID":"a5dca30e-711d-43e1-99c6-32c146cc324a","Type":"ContainerStarted","Data":"3248ec05b074f1a6c41266f205ab8c6728a148f5f1e98b733e45449ad9f804c1"} Jul 10 17:51:40.908783 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:40.908725 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" podStartSLOduration=1.908695871 podStartE2EDuration="1.908695871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:51:40.907370018 +0000 UTC m=+2426.976567120" watchObservedRunningTime="2026-07-10 17:51:40.908695871 +0000 UTC m=+2426.977892971" Jul 10 17:51:45.052946 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:45.052911 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:51:50.168350 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:50.168318 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:50.168742 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:50.168394 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:50.180772 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:50.180749 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:50.952010 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:50.951978 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:51:59.607781 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:51:59.607749 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:07.327036 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:07.326998 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:09.561926 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:09.561891 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:52:09.782677 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:09.782635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:28.455047 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:28.455015 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:52:34.374248 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:34.374207 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj"] Jul 10 17:52:34.387300 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:34.387272 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:35.244482 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:35.244447 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw"] Jul 10 17:52:35.269770 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:35.269738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:44.551762 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.551728 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv"] Jul 10 17:52:44.555202 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.555185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.557657 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.557629 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3beb3b\"/\"rate-test-model-3beb3b-kserve-self-signed-certs\"" Jul 10 17:52:44.572762 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.572735 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv"] Jul 10 17:52:44.591905 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.591875 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:44.662926 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.662887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-model-cache\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.663104 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.662978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tmp-dir\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.663104 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.663034 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gncm\" (UniqueName: \"kubernetes.io/projected/f6bb6490-6bcd-4d7b-9c3e-51438601da43-kube-api-access-5gncm\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.663104 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.663061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-home\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.663104 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.663090 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tls-certs\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.663248 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.663126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-dshm\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.763661 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.763631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tmp-dir\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.763873 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.763669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gncm\" (UniqueName: \"kubernetes.io/projected/f6bb6490-6bcd-4d7b-9c3e-51438601da43-kube-api-access-5gncm\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.763873 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.763693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-home\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.763873 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.763753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tls-certs\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.763873 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.763855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-dshm\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.764109 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.763926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-model-cache\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.764109 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.764062 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tmp-dir\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.764210 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.764126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-home\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.764264 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.764244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-model-cache\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.766040 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.766020 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-dshm\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.766240 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.766227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tls-certs\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.771421 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.771399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gncm\" (UniqueName: \"kubernetes.io/projected/f6bb6490-6bcd-4d7b-9c3e-51438601da43-kube-api-access-5gncm\") pod \"rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:44.864696 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:44.864625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:45.000162 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:45.000138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv"] Jul 10 17:52:45.002306 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:52:45.002279 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6bb6490_6bcd_4d7b_9c3e_51438601da43.slice/crio-e753998ecbd0957d4db29249af069f804143375d18b6e27dbda26326c0e982b8 WatchSource:0}: Error finding container e753998ecbd0957d4db29249af069f804143375d18b6e27dbda26326c0e982b8: Status 404 returned error can't find the container with id e753998ecbd0957d4db29249af069f804143375d18b6e27dbda26326c0e982b8 Jul 10 17:52:45.198383 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:45.198347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" event={"ID":"f6bb6490-6bcd-4d7b-9c3e-51438601da43","Type":"ContainerStarted","Data":"c549285173260b14370abbdfd535b2a03b814ff5884c7f34a77bcb626a072444"} Jul 10 17:52:45.198383 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:45.198394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" event={"ID":"f6bb6490-6bcd-4d7b-9c3e-51438601da43","Type":"ContainerStarted","Data":"e753998ecbd0957d4db29249af069f804143375d18b6e27dbda26326c0e982b8"} Jul 10 17:52:45.216129 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:45.216080 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" podStartSLOduration=1.216065578 podStartE2EDuration="1.216065578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:52:45.215371298 +0000 UTC m=+2491.284568397" watchObservedRunningTime="2026-07-10 17:52:45.216065578 +0000 UTC m=+2491.285262678" Jul 10 17:52:48.350600 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:48.350565 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:52:49.394434 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.394397 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" podUID="4e08686f-1b42-421f-baad-74610cede365" containerName="main" containerID="cri-o://4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8" gracePeriod=45 Jul 10 17:52:49.650996 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.650939 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:52:49.707358 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707330 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtwzx\" (UniqueName: \"kubernetes.io/projected/4e08686f-1b42-421f-baad-74610cede365-kube-api-access-qtwzx\") pod \"4e08686f-1b42-421f-baad-74610cede365\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " Jul 10 17:52:49.707522 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707366 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-model-cache\") pod \"4e08686f-1b42-421f-baad-74610cede365\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " Jul 10 17:52:49.707522 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707395 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-dshm\") pod \"4e08686f-1b42-421f-baad-74610cede365\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " Jul 10 17:52:49.707522 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707436 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-home\") pod \"4e08686f-1b42-421f-baad-74610cede365\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " Jul 10 17:52:49.707522 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707513 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-tmp-dir\") pod \"4e08686f-1b42-421f-baad-74610cede365\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " Jul 10 17:52:49.707696 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707538 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e08686f-1b42-421f-baad-74610cede365-tls-certs\") pod \"4e08686f-1b42-421f-baad-74610cede365\" (UID: \"4e08686f-1b42-421f-baad-74610cede365\") " Jul 10 17:52:49.707696 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707632 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e08686f-1b42-421f-baad-74610cede365" (UID: "4e08686f-1b42-421f-baad-74610cede365"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:52:49.707814 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707781 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-home" (OuterVolumeSpecName: "home") pod "4e08686f-1b42-421f-baad-74610cede365" (UID: "4e08686f-1b42-421f-baad-74610cede365"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:52:49.707871 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707853 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e08686f-1b42-421f-baad-74610cede365" (UID: "4e08686f-1b42-421f-baad-74610cede365"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:52:49.707940 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707920 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:49.707995 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.707943 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:49.709531 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.709502 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-dshm" (OuterVolumeSpecName: "dshm") pod "4e08686f-1b42-421f-baad-74610cede365" (UID: "4e08686f-1b42-421f-baad-74610cede365"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:52:49.709531 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.709519 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e08686f-1b42-421f-baad-74610cede365-kube-api-access-qtwzx" (OuterVolumeSpecName: "kube-api-access-qtwzx") pod "4e08686f-1b42-421f-baad-74610cede365" (UID: "4e08686f-1b42-421f-baad-74610cede365"). InnerVolumeSpecName "kube-api-access-qtwzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:52:49.709674 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.709561 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e08686f-1b42-421f-baad-74610cede365-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e08686f-1b42-421f-baad-74610cede365" (UID: "4e08686f-1b42-421f-baad-74610cede365"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:52:49.809202 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.809169 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:49.809202 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.809198 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e08686f-1b42-421f-baad-74610cede365-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:49.809411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.809211 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwzx\" (UniqueName: \"kubernetes.io/projected/4e08686f-1b42-421f-baad-74610cede365-kube-api-access-qtwzx\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:49.809411 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:49.809224 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e08686f-1b42-421f-baad-74610cede365-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:50.225492 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.225457 2580 generic.go:356] "Generic (PLEG): container finished" podID="4e08686f-1b42-421f-baad-74610cede365" containerID="4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8" exitCode=0 Jul 10 17:52:50.225652 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.225505 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" Jul 10 17:52:50.225652 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.225539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" event={"ID":"4e08686f-1b42-421f-baad-74610cede365","Type":"ContainerDied","Data":"4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8"} Jul 10 17:52:50.225652 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.225583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj" event={"ID":"4e08686f-1b42-421f-baad-74610cede365","Type":"ContainerDied","Data":"1833018778d100af5dff19cd388e107018c605c59f192343ba052e5b50198daa"} Jul 10 17:52:50.225652 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.225602 2580 scope.go:117] "RemoveContainer" containerID="4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8" Jul 10 17:52:50.234539 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.234522 2580 scope.go:117] "RemoveContainer" containerID="4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8" Jul 10 17:52:50.234818 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:52:50.234801 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8\": container with ID starting with 4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8 not found: ID does not exist" containerID="4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8" Jul 10 17:52:50.234879 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.234826 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8"} err="failed to get container status \"4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8\": rpc error: code = NotFound desc = could not find container \"4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8\": container with ID starting with 4f078ccf2fe4b4837712e5e786b48b7cc2cba18c46b72dcda589a0fc30cf6ee8 not found: ID does not exist" Jul 10 17:52:50.246279 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.246258 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj"] Jul 10 17:52:50.249203 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.249186 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/sub-test-model-3beb3b-kserve-988dff94f-qpdwj"] Jul 10 17:52:50.264372 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.264346 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" podUID="a5dca30e-711d-43e1-99c6-32c146cc324a" containerName="main" containerID="cri-o://ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8" gracePeriod=45 Jul 10 17:52:50.503375 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.503344 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e08686f-1b42-421f-baad-74610cede365" path="/var/lib/kubelet/pods/4e08686f-1b42-421f-baad-74610cede365/volumes" Jul 10 17:52:50.518534 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.518515 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:52:50.616865 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.616830 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-model-cache\") pod \"a5dca30e-711d-43e1-99c6-32c146cc324a\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " Jul 10 17:52:50.617057 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.616878 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-dshm\") pod \"a5dca30e-711d-43e1-99c6-32c146cc324a\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " Jul 10 17:52:50.617057 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.616942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-tmp-dir\") pod \"a5dca30e-711d-43e1-99c6-32c146cc324a\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " Jul 10 17:52:50.617057 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.616971 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dca30e-711d-43e1-99c6-32c146cc324a-tls-certs\") pod \"a5dca30e-711d-43e1-99c6-32c146cc324a\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " Jul 10 17:52:50.617210 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.617073 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zs5f\" (UniqueName: \"kubernetes.io/projected/a5dca30e-711d-43e1-99c6-32c146cc324a-kube-api-access-9zs5f\") pod \"a5dca30e-711d-43e1-99c6-32c146cc324a\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " Jul 10 17:52:50.617210 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.617152 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5dca30e-711d-43e1-99c6-32c146cc324a" (UID: "a5dca30e-711d-43e1-99c6-32c146cc324a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:52:50.617315 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.617230 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-home\") pod \"a5dca30e-711d-43e1-99c6-32c146cc324a\" (UID: \"a5dca30e-711d-43e1-99c6-32c146cc324a\") " Jul 10 17:52:50.617315 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.617244 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5dca30e-711d-43e1-99c6-32c146cc324a" (UID: "a5dca30e-711d-43e1-99c6-32c146cc324a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:52:50.617490 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.617467 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-home" (OuterVolumeSpecName: "home") pod "a5dca30e-711d-43e1-99c6-32c146cc324a" (UID: "a5dca30e-711d-43e1-99c6-32c146cc324a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:52:50.617571 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.617556 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:50.617630 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.617600 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:50.617630 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.617614 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:50.619146 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.619123 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5dca30e-711d-43e1-99c6-32c146cc324a-kube-api-access-9zs5f" (OuterVolumeSpecName: "kube-api-access-9zs5f") pod "a5dca30e-711d-43e1-99c6-32c146cc324a" (UID: "a5dca30e-711d-43e1-99c6-32c146cc324a"). InnerVolumeSpecName "kube-api-access-9zs5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:52:50.619503 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.619482 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-dshm" (OuterVolumeSpecName: "dshm") pod "a5dca30e-711d-43e1-99c6-32c146cc324a" (UID: "a5dca30e-711d-43e1-99c6-32c146cc324a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:52:50.619576 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.619485 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5dca30e-711d-43e1-99c6-32c146cc324a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5dca30e-711d-43e1-99c6-32c146cc324a" (UID: "a5dca30e-711d-43e1-99c6-32c146cc324a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:52:50.719051 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.719020 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dca30e-711d-43e1-99c6-32c146cc324a-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:50.719051 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.719046 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dca30e-711d-43e1-99c6-32c146cc324a-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:50.719205 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:50.719061 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zs5f\" (UniqueName: \"kubernetes.io/projected/a5dca30e-711d-43e1-99c6-32c146cc324a-kube-api-access-9zs5f\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:52:51.231990 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:51.231957 2580 generic.go:356] "Generic (PLEG): container finished" podID="a5dca30e-711d-43e1-99c6-32c146cc324a" containerID="ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8" exitCode=0 Jul 10 17:52:51.232186 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:51.232014 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" Jul 10 17:52:51.232186 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:51.232038 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" event={"ID":"a5dca30e-711d-43e1-99c6-32c146cc324a","Type":"ContainerDied","Data":"ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8"} Jul 10 17:52:51.232186 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:51.232078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw" event={"ID":"a5dca30e-711d-43e1-99c6-32c146cc324a","Type":"ContainerDied","Data":"3248ec05b074f1a6c41266f205ab8c6728a148f5f1e98b733e45449ad9f804c1"} Jul 10 17:52:51.232186 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:51.232100 2580 scope.go:117] "RemoveContainer" containerID="ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8" Jul 10 17:52:51.240926 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:51.240909 2580 scope.go:117] "RemoveContainer" containerID="ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8" Jul 10 17:52:51.241185 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:52:51.241166 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8\": container with ID starting with ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8 not found: ID does not exist" containerID="ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8" Jul 10 17:52:51.241236 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:51.241195 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8"} err="failed to get container status \"ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8\": rpc error: code = NotFound desc = could not find container \"ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8\": container with ID starting with ef0e0a4c9cb4b5f5e793e5b014dadc69ea9aca478e8094f53afd68f08a57c2e8 not found: ID does not exist" Jul 10 17:52:51.253502 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:51.253478 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw"] Jul 10 17:52:51.258851 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:51.258829 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/sub-test-model-9f66f0-kserve-5fdfcc965-pq4kw"] Jul 10 17:52:52.502135 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:52.502107 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5dca30e-711d-43e1-99c6-32c146cc324a" path="/var/lib/kubelet/pods/a5dca30e-711d-43e1-99c6-32c146cc324a/volumes" Jul 10 17:52:54.865173 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:54.865136 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:54.865173 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:54.865180 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:54.877696 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:54.877672 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:52:55.264476 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:52:55.264447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:53:05.049904 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.049867 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t"] Jul 10 17:53:05.050357 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.050292 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5dca30e-711d-43e1-99c6-32c146cc324a" containerName="main" Jul 10 17:53:05.050357 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.050306 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5dca30e-711d-43e1-99c6-32c146cc324a" containerName="main" Jul 10 17:53:05.050357 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.050317 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e08686f-1b42-421f-baad-74610cede365" containerName="main" Jul 10 17:53:05.050357 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.050323 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e08686f-1b42-421f-baad-74610cede365" containerName="main" Jul 10 17:53:05.050489 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.050378 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e08686f-1b42-421f-baad-74610cede365" containerName="main" Jul 10 17:53:05.050489 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.050386 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5dca30e-711d-43e1-99c6-32c146cc324a" containerName="main" Jul 10 17:53:05.054700 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.054682 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.057480 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.057457 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"openshift-service-ca.crt\"" Jul 10 17:53:05.057480 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.057477 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"kube-root-ca.crt\"" Jul 10 17:53:05.058468 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.058451 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"default-dockercfg-7td5b\"" Jul 10 17:53:05.058558 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.058496 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9f66f0\"/\"rate-test-model-9f66f0-kserve-self-signed-certs\"" Jul 10 17:53:05.061928 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.061795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t"] Jul 10 17:53:05.144817 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.144785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:05.147286 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.147259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-dshm\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.147432 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.147311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tls-certs\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.147432 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.147394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ksq7\" (UniqueName: \"kubernetes.io/projected/2eea90a8-e517-4c7b-9bad-76b847edfbdf-kube-api-access-4ksq7\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.147575 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.147439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-home\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.147575 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.147472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-model-cache\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.147575 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.147559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tmp-dir\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.248509 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.248470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-dshm\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.248509 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.248518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tls-certs\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.248779 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.248551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ksq7\" (UniqueName: \"kubernetes.io/projected/2eea90a8-e517-4c7b-9bad-76b847edfbdf-kube-api-access-4ksq7\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.248779 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.248574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-home\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.248779 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.248610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-model-cache\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.248779 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.248754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tmp-dir\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.249001 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.248951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-home\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.249058 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.249044 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-model-cache\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.249165 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.249143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tmp-dir\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.250819 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.250799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-dshm\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.251005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.250987 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tls-certs\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.255539 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.255517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ksq7\" (UniqueName: \"kubernetes.io/projected/2eea90a8-e517-4c7b-9bad-76b847edfbdf-kube-api-access-4ksq7\") pod \"rate-test-model-9f66f0-kserve-75ff86559-tl78t\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.364651 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.364586 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:05.491144 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:05.491121 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t"] Jul 10 17:53:05.493610 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:53:05.493577 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eea90a8_e517_4c7b_9bad_76b847edfbdf.slice/crio-f0c5767398a74c1e3f4474fe92c5dd7d2de152803ea53eebfe5964d557f75c9a WatchSource:0}: Error finding container f0c5767398a74c1e3f4474fe92c5dd7d2de152803ea53eebfe5964d557f75c9a: Status 404 returned error can't find the container with id f0c5767398a74c1e3f4474fe92c5dd7d2de152803ea53eebfe5964d557f75c9a Jul 10 17:53:06.307959 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:06.307878 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" event={"ID":"2eea90a8-e517-4c7b-9bad-76b847edfbdf","Type":"ContainerStarted","Data":"a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b"} Jul 10 17:53:06.307959 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:06.307922 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" event={"ID":"2eea90a8-e517-4c7b-9bad-76b847edfbdf","Type":"ContainerStarted","Data":"f0c5767398a74c1e3f4474fe92c5dd7d2de152803ea53eebfe5964d557f75c9a"} Jul 10 17:53:06.323944 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:06.323901 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" podStartSLOduration=1.323886691 podStartE2EDuration="1.323886691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:53:06.323429198 +0000 UTC m=+2512.392626299" watchObservedRunningTime="2026-07-10 17:53:06.323886691 +0000 UTC m=+2512.393083770" Jul 10 17:53:10.558172 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:10.558138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:53:15.364801 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:15.364766 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:15.365187 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:15.364817 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:15.377360 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:15.377333 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:16.367744 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:16.367690 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:53:24.955106 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:24.955067 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:53:26.897535 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:26.897501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:26.998835 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:26.998801 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:33.691636 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:33.691597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:33.791697 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:33.791662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:45.254095 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:45.254054 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:53:49.440179 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:49.440149 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv"] Jul 10 17:53:49.459682 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:49.459657 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:50.311236 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:50.311196 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t"] Jul 10 17:53:50.356723 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:53:50.356674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:01.546681 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:01.546638 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:54:04.460003 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.459958 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" podUID="f6bb6490-6bcd-4d7b-9c3e-51438601da43" containerName="main" containerID="cri-o://c549285173260b14370abbdfd535b2a03b814ff5884c7f34a77bcb626a072444" gracePeriod=45 Jul 10 17:54:04.596163 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.596132 2580 generic.go:356] "Generic (PLEG): container finished" podID="f6bb6490-6bcd-4d7b-9c3e-51438601da43" containerID="c549285173260b14370abbdfd535b2a03b814ff5884c7f34a77bcb626a072444" exitCode=0 Jul 10 17:54:04.596292 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.596199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" event={"ID":"f6bb6490-6bcd-4d7b-9c3e-51438601da43","Type":"ContainerDied","Data":"c549285173260b14370abbdfd535b2a03b814ff5884c7f34a77bcb626a072444"} Jul 10 17:54:04.708955 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.708932 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:54:04.783234 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783167 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-model-cache\") pod \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " Jul 10 17:54:04.783234 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783231 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-dshm\") pod \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " Jul 10 17:54:04.783434 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783278 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-home\") pod \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " Jul 10 17:54:04.783434 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783308 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gncm\" (UniqueName: \"kubernetes.io/projected/f6bb6490-6bcd-4d7b-9c3e-51438601da43-kube-api-access-5gncm\") pod \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " Jul 10 17:54:04.783434 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783345 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tmp-dir\") pod \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " Jul 10 17:54:04.783434 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783363 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tls-certs\") pod \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\" (UID: \"f6bb6490-6bcd-4d7b-9c3e-51438601da43\") " Jul 10 17:54:04.783434 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783405 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6bb6490-6bcd-4d7b-9c3e-51438601da43" (UID: "f6bb6490-6bcd-4d7b-9c3e-51438601da43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:54:04.783678 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783535 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-home" (OuterVolumeSpecName: "home") pod "f6bb6490-6bcd-4d7b-9c3e-51438601da43" (UID: "f6bb6490-6bcd-4d7b-9c3e-51438601da43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:54:04.783678 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783618 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6bb6490-6bcd-4d7b-9c3e-51438601da43" (UID: "f6bb6490-6bcd-4d7b-9c3e-51438601da43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:54:04.783678 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783671 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:04.783821 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.783683 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:04.785459 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.785439 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-dshm" (OuterVolumeSpecName: "dshm") pod "f6bb6490-6bcd-4d7b-9c3e-51438601da43" (UID: "f6bb6490-6bcd-4d7b-9c3e-51438601da43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:54:04.785529 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.785467 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bb6490-6bcd-4d7b-9c3e-51438601da43-kube-api-access-5gncm" (OuterVolumeSpecName: "kube-api-access-5gncm") pod "f6bb6490-6bcd-4d7b-9c3e-51438601da43" (UID: "f6bb6490-6bcd-4d7b-9c3e-51438601da43"). InnerVolumeSpecName "kube-api-access-5gncm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:54:04.785529 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.785506 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6bb6490-6bcd-4d7b-9c3e-51438601da43" (UID: "f6bb6490-6bcd-4d7b-9c3e-51438601da43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:54:04.787882 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.787861 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:04.884140 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.884110 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:04.884140 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.884135 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bb6490-6bcd-4d7b-9c3e-51438601da43-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:04.884140 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.884143 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bb6490-6bcd-4d7b-9c3e-51438601da43-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:04.884337 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.884151 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gncm\" (UniqueName: \"kubernetes.io/projected/f6bb6490-6bcd-4d7b-9c3e-51438601da43-kube-api-access-5gncm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:04.888625 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:04.888603 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:05.330851 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.330810 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" podUID="2eea90a8-e517-4c7b-9bad-76b847edfbdf" containerName="main" containerID="cri-o://a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b" gracePeriod=45 Jul 10 17:54:05.576956 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.576926 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:54:05.602511 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.602451 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" Jul 10 17:54:05.602511 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.602479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv" event={"ID":"f6bb6490-6bcd-4d7b-9c3e-51438601da43","Type":"ContainerDied","Data":"e753998ecbd0957d4db29249af069f804143375d18b6e27dbda26326c0e982b8"} Jul 10 17:54:05.602697 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.602515 2580 scope.go:117] "RemoveContainer" containerID="c549285173260b14370abbdfd535b2a03b814ff5884c7f34a77bcb626a072444" Jul 10 17:54:05.604005 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.603968 2580 generic.go:356] "Generic (PLEG): container finished" podID="2eea90a8-e517-4c7b-9bad-76b847edfbdf" containerID="a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b" exitCode=0 Jul 10 17:54:05.604120 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.604013 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" Jul 10 17:54:05.604120 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.604009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" event={"ID":"2eea90a8-e517-4c7b-9bad-76b847edfbdf","Type":"ContainerDied","Data":"a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b"} Jul 10 17:54:05.604239 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.604134 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t" event={"ID":"2eea90a8-e517-4c7b-9bad-76b847edfbdf","Type":"ContainerDied","Data":"f0c5767398a74c1e3f4474fe92c5dd7d2de152803ea53eebfe5964d557f75c9a"} Jul 10 17:54:05.611929 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.611913 2580 scope.go:117] "RemoveContainer" containerID="a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b" Jul 10 17:54:05.620970 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.620947 2580 scope.go:117] "RemoveContainer" containerID="a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b" Jul 10 17:54:05.621582 ip-10-0-140-169 kubenswrapper[2580]: E0710 17:54:05.621558 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b\": container with ID starting with a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b not found: ID does not exist" containerID="a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b" Jul 10 17:54:05.621683 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.621588 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b"} err="failed to get container status \"a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b\": rpc error: code = NotFound desc = could not find container \"a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b\": container with ID starting with a89ec9dc21c645dccfb741293aed79610cc17316d2b4339b24d83ea2def5a37b not found: ID does not exist" Jul 10 17:54:05.623426 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.623406 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv"] Jul 10 17:54:05.626490 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.626471 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3beb3b/rate-test-model-3beb3b-kserve-9c9b4b5f-25mkv"] Jul 10 17:54:05.690410 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.690390 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tls-certs\") pod \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " Jul 10 17:54:05.690543 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.690437 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-dshm\") pod \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " Jul 10 17:54:05.690543 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.690485 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ksq7\" (UniqueName: \"kubernetes.io/projected/2eea90a8-e517-4c7b-9bad-76b847edfbdf-kube-api-access-4ksq7\") pod \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " Jul 10 17:54:05.690543 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.690518 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-model-cache\") pod \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " Jul 10 17:54:05.690670 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.690592 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tmp-dir\") pod \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " Jul 10 17:54:05.690755 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.690688 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-home\") pod \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\" (UID: \"2eea90a8-e517-4c7b-9bad-76b847edfbdf\") " Jul 10 17:54:05.690921 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.690895 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2eea90a8-e517-4c7b-9bad-76b847edfbdf" (UID: "2eea90a8-e517-4c7b-9bad-76b847edfbdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:54:05.690921 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.690894 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-model-cache" (OuterVolumeSpecName: "model-cache") pod "2eea90a8-e517-4c7b-9bad-76b847edfbdf" (UID: "2eea90a8-e517-4c7b-9bad-76b847edfbdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:54:05.691066 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.690947 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-home" (OuterVolumeSpecName: "home") pod "2eea90a8-e517-4c7b-9bad-76b847edfbdf" (UID: "2eea90a8-e517-4c7b-9bad-76b847edfbdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:54:05.691146 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.691127 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:05.691184 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.691153 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:05.691184 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.691169 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:05.692526 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.692505 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-dshm" (OuterVolumeSpecName: "dshm") pod "2eea90a8-e517-4c7b-9bad-76b847edfbdf" (UID: "2eea90a8-e517-4c7b-9bad-76b847edfbdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:54:05.692526 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.692518 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eea90a8-e517-4c7b-9bad-76b847edfbdf-kube-api-access-4ksq7" (OuterVolumeSpecName: "kube-api-access-4ksq7") pod "2eea90a8-e517-4c7b-9bad-76b847edfbdf" (UID: "2eea90a8-e517-4c7b-9bad-76b847edfbdf"). InnerVolumeSpecName "kube-api-access-4ksq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:54:05.692625 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.692575 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2eea90a8-e517-4c7b-9bad-76b847edfbdf" (UID: "2eea90a8-e517-4c7b-9bad-76b847edfbdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:54:05.791870 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.791827 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eea90a8-e517-4c7b-9bad-76b847edfbdf-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:05.791870 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.791866 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eea90a8-e517-4c7b-9bad-76b847edfbdf-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:05.791870 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.791875 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ksq7\" (UniqueName: \"kubernetes.io/projected/2eea90a8-e517-4c7b-9bad-76b847edfbdf-kube-api-access-4ksq7\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\"" Jul 10 17:54:05.924446 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.924419 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t"] Jul 10 17:54:05.926498 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:05.926478 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9f66f0/rate-test-model-9f66f0-kserve-75ff86559-tl78t"] Jul 10 17:54:06.508113 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:06.508078 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eea90a8-e517-4c7b-9bad-76b847edfbdf" path="/var/lib/kubelet/pods/2eea90a8-e517-4c7b-9bad-76b847edfbdf/volumes" Jul 10 17:54:06.508466 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:06.508452 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6bb6490-6bcd-4d7b-9c3e-51438601da43" path="/var/lib/kubelet/pods/f6bb6490-6bcd-4d7b-9c3e-51438601da43/volumes" Jul 10 17:54:09.911317 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:09.911283 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:10.010567 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:10.010537 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:19.975670 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:19.975629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:22.200995 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:22.200951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:23.180963 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:23.180928 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:54:24.485588 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:24.485541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:30.587753 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:30.587693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:30.662744 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:30.662668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:35.153868 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:35.153818 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:35.375259 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:35.375220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:35.991258 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:35.991177 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:36.092420 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:36.092363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:45.914269 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:45.914224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:49.154997 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:49.154952 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g7dkh"] Jul 10 17:54:50.559087 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:50.559043 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:56.389441 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:56.389407 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:56.492431 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:54:56.492397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:01.266685 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:01.266642 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:01.488198 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:01.488164 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:01.924641 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:01.924611 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-764b77c78b-577vn_19426143-0472-46ad-b7e6-1e09bb828f86/authorino/0.log" Jul 10 17:55:07.425550 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:07.425516 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b884cfc77-d8l8z_fbead02a-0a11-42c8-a84d-d59c13dbb822/manager/0.log" Jul 10 17:55:07.431565 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:07.431543 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b884cfc77-d8l8z_fbead02a-0a11-42c8-a84d-d59c13dbb822/copy-manifests/0.log" Jul 10 17:55:10.813428 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:10.813400 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-764b77c78b-577vn_19426143-0472-46ad-b7e6-1e09bb828f86/authorino/0.log" Jul 10 17:55:11.160325 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:11.160298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cv5nk_808e3317-6dd7-4dd0-bb97-152a393f3cbe/kuadrant-console-plugin/0.log" Jul 10 17:55:11.495782 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:11.495685 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g7dkh_9f3ef799-d4b8-490e-98a0-7a2418b253fa/limitador/0.log" Jul 10 17:55:12.198504 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:12.198474 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545f68d885-h8rwn_d4db34a1-8704-4891-a9a6-5ee9e8108291/kube-auth-proxy/0.log" Jul 10 17:55:13.531579 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:13.531551 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-sb8z2_d41fcfde-8f83-4209-b40a-b4aa21fa930d/main/0.log" Jul 10 17:55:13.638873 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:13.638841 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-2jsmn_8d21842c-61ed-4015-aa8b-9d5f69a3fd4c/main/0.log" Jul 10 17:55:17.253494 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:17.253466 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:17.350442 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:17.350419 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:19.942624 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:19.942592 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pw595_b9842d85-1f4d-4c40-a1ff-2254160fdb9e/global-pull-secret-syncer/0.log" Jul 10 17:55:20.034314 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:20.034288 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4874w_9a301d47-a92b-46d1-aaf9-08ecf8ffccd6/konnectivity-agent/0.log" Jul 10 17:55:20.193308 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:20.193210 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-169.ec2.internal_27bd9ee91ed58c4c427c2a52d0f7a6ad/haproxy/0.log" Jul 10 17:55:24.829781 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:24.829730 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-764b77c78b-577vn_19426143-0472-46ad-b7e6-1e09bb828f86/authorino/0.log" Jul 10 17:55:24.927669 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:24.927591 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cv5nk_808e3317-6dd7-4dd0-bb97-152a393f3cbe/kuadrant-console-plugin/0.log" Jul 10 17:55:25.218198 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:25.218120 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g7dkh_9f3ef799-d4b8-490e-98a0-7a2418b253fa/limitador/0.log" Jul 10 17:55:26.886767 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:26.886740 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3/alertmanager/0.log" Jul 10 17:55:26.913402 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:26.913379 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3/config-reloader/0.log" Jul 10 17:55:26.935251 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:26.935230 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3/kube-rbac-proxy-web/0.log" Jul 10 17:55:26.957621 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:26.957599 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3/kube-rbac-proxy/0.log" Jul 10 17:55:26.978566 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:26.978549 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3/kube-rbac-proxy-metric/0.log" Jul 10 17:55:27.000760 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.000683 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3/prom-label-proxy/0.log" Jul 10 17:55:27.026738 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.026692 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f86261f9-6ae2-4ac6-a8e3-7473d42c7cf3/init-config-reloader/0.log" Jul 10 17:55:27.120434 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.120407 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-42jvt_c971946f-e6b7-4b3e-9d1c-54109d5dbd86/kube-state-metrics/0.log" Jul 10 17:55:27.139482 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.139458 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-42jvt_c971946f-e6b7-4b3e-9d1c-54109d5dbd86/kube-rbac-proxy-main/0.log" Jul 10 17:55:27.160311 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.160289 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-42jvt_c971946f-e6b7-4b3e-9d1c-54109d5dbd86/kube-rbac-proxy-self/0.log" Jul 10 17:55:27.188448 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.188428 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-645c996df6-cqz6v_7e1f7706-6282-4803-861d-82728a66b357/metrics-server/0.log" Jul 10 17:55:27.213332 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.213312 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-bqmrj_99399577-bfd1-49b8-bb71-6651807c4975/monitoring-plugin/0.log" Jul 10 17:55:27.340457 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.340437 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nhfq9_f68bc723-68a6-4a35-9cf9-4c79895b02cd/node-exporter/0.log" Jul 10 17:55:27.365682 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.365661 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nhfq9_f68bc723-68a6-4a35-9cf9-4c79895b02cd/kube-rbac-proxy/0.log" Jul 10 17:55:27.390205 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.390178 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nhfq9_f68bc723-68a6-4a35-9cf9-4c79895b02cd/init-textfile/0.log" Jul 10 17:55:27.562596 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.562566 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tdsvt_accfc9e9-b413-4f75-bcbe-eae0da6c8370/kube-rbac-proxy-main/0.log" Jul 10 17:55:27.584337 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.584315 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tdsvt_accfc9e9-b413-4f75-bcbe-eae0da6c8370/kube-rbac-proxy-self/0.log" Jul 10 17:55:27.605063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.605012 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tdsvt_accfc9e9-b413-4f75-bcbe-eae0da6c8370/openshift-state-metrics/0.log" Jul 10 17:55:27.642311 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.642293 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2416a648-be17-44aa-84cc-85a734c114d7/prometheus/0.log" Jul 10 17:55:27.668624 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.668607 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2416a648-be17-44aa-84cc-85a734c114d7/config-reloader/0.log" Jul 10 17:55:27.690114 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.690099 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2416a648-be17-44aa-84cc-85a734c114d7/thanos-sidecar/0.log" Jul 10 17:55:27.711283 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.711267 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2416a648-be17-44aa-84cc-85a734c114d7/kube-rbac-proxy-web/0.log" Jul 10 17:55:27.734234 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.734218 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2416a648-be17-44aa-84cc-85a734c114d7/kube-rbac-proxy/0.log" Jul 10 17:55:27.762982 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.762961 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2416a648-be17-44aa-84cc-85a734c114d7/kube-rbac-proxy-thanos/0.log" Jul 10 17:55:27.783396 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.783373 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2416a648-be17-44aa-84cc-85a734c114d7/init-config-reloader/0.log" Jul 10 17:55:27.811307 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.811287 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-8ch2h_860399f4-6cc7-44bb-95bd-60e1b92e5d81/prometheus-operator/0.log" Jul 10 17:55:27.831556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.831530 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-8ch2h_860399f4-6cc7-44bb-95bd-60e1b92e5d81/kube-rbac-proxy/0.log" Jul 10 17:55:27.900798 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.900777 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-955bdfdbb-8gct9_23106896-b575-4d42-9c9e-1bd865ff1771/telemeter-client/0.log" Jul 10 17:55:27.923285 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.923268 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-955bdfdbb-8gct9_23106896-b575-4d42-9c9e-1bd865ff1771/reload/0.log" Jul 10 17:55:27.946477 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.946457 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-955bdfdbb-8gct9_23106896-b575-4d42-9c9e-1bd865ff1771/kube-rbac-proxy/0.log" Jul 10 17:55:27.980273 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:27.980253 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f899fbcd8-t77w7_4caec5d7-6528-4315-9b54-469e3f87ab96/thanos-query/0.log" Jul 10 17:55:28.004779 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:28.004762 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f899fbcd8-t77w7_4caec5d7-6528-4315-9b54-469e3f87ab96/kube-rbac-proxy-web/0.log" Jul 10 17:55:28.028297 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:28.028280 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f899fbcd8-t77w7_4caec5d7-6528-4315-9b54-469e3f87ab96/kube-rbac-proxy/0.log" Jul 10 17:55:28.051215 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:28.051197 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f899fbcd8-t77w7_4caec5d7-6528-4315-9b54-469e3f87ab96/prom-label-proxy/0.log" Jul 10 17:55:28.077102 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:28.077086 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f899fbcd8-t77w7_4caec5d7-6528-4315-9b54-469e3f87ab96/kube-rbac-proxy-rules/0.log" Jul 10 17:55:28.097584 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:28.097566 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f899fbcd8-t77w7_4caec5d7-6528-4315-9b54-469e3f87ab96/kube-rbac-proxy-metrics/0.log" Jul 10 17:55:30.451332 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.451294 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z"] Jul 10 17:55:30.451899 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.451875 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6bb6490-6bcd-4d7b-9c3e-51438601da43" containerName="main" Jul 10 17:55:30.451899 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.451901 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6bb6490-6bcd-4d7b-9c3e-51438601da43" containerName="main" Jul 10 17:55:30.452063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.451912 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eea90a8-e517-4c7b-9bad-76b847edfbdf" containerName="main" Jul 10 17:55:30.452063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.451921 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eea90a8-e517-4c7b-9bad-76b847edfbdf" containerName="main" Jul 10 17:55:30.452063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.452012 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eea90a8-e517-4c7b-9bad-76b847edfbdf" containerName="main" Jul 10 17:55:30.452063 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.452027 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6bb6490-6bcd-4d7b-9c3e-51438601da43" containerName="main" Jul 10 17:55:30.455077 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.455058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.457625 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.457604 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vnztt\"/\"openshift-service-ca.crt\"" Jul 10 17:55:30.458610 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.458589 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vnztt\"/\"kube-root-ca.crt\"" Jul 10 17:55:30.458754 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.458589 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vnztt\"/\"default-dockercfg-6c996\"" Jul 10 17:55:30.465560 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.465536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z"] Jul 10 17:55:30.552129 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.552108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-sys\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.552129 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.552135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-982fs\" (UniqueName: \"kubernetes.io/projected/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-kube-api-access-982fs\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.552340 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.552171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-podres\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.552340 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.552242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-proc\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.552446 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.552361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-lib-modules\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.653409 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.653383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-lib-modules\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.653522 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.653440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-sys\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.653522 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.653501 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-sys\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.653604 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.653532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-982fs\" (UniqueName: \"kubernetes.io/projected/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-kube-api-access-982fs\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.653604 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.653565 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-lib-modules\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.653604 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.653572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-podres\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.653741 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.653621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-proc\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.653741 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.653639 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-podres\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.653741 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.653682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-proc\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.662341 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.662319 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-982fs\" (UniqueName: \"kubernetes.io/projected/1aa18205-9fb4-4f07-8a79-1ba4e5e85edd-kube-api-access-982fs\") pod \"perf-node-gather-daemonset-jsx2z\" (UID: \"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.766064 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.765976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:30.894558 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:30.894526 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z"] Jul 10 17:55:30.897616 ip-10-0-140-169 kubenswrapper[2580]: W0710 17:55:30.897587 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1aa18205_9fb4_4f07_8a79_1ba4e5e85edd.slice/crio-8f29b7ce9411d92f7142c84f1a116eb553701cc34c56a4670e71fcdd5b222725 WatchSource:0}: Error finding container 8f29b7ce9411d92f7142c84f1a116eb553701cc34c56a4670e71fcdd5b222725: Status 404 returned error can't find the container with id 8f29b7ce9411d92f7142c84f1a116eb553701cc34c56a4670e71fcdd5b222725 Jul 10 17:55:31.003642 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:31.003616 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" event={"ID":"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd","Type":"ContainerStarted","Data":"8f29b7ce9411d92f7142c84f1a116eb553701cc34c56a4670e71fcdd5b222725"} Jul 10 17:55:32.010903 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:32.010868 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" event={"ID":"1aa18205-9fb4-4f07-8a79-1ba4e5e85edd","Type":"ContainerStarted","Data":"f66029547387ea28a468c13d01c82f0ea0dc54de813bebe570977dcd4ab4f2f4"} Jul 10 17:55:32.011299 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:32.011003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:32.030650 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:32.030595 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" podStartSLOduration=2.030575504 podStartE2EDuration="2.030575504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:55:32.029529154 +0000 UTC m=+2658.098726254" watchObservedRunningTime="2026-07-10 17:55:32.030575504 +0000 UTC m=+2658.099772605" Jul 10 17:55:32.364129 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:32.364051 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jg4wj_39c353e2-486c-4f5d-a367-7a77b46fd9ba/dns/0.log" Jul 10 17:55:32.384991 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:32.384964 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jg4wj_39c353e2-486c-4f5d-a367-7a77b46fd9ba/kube-rbac-proxy/0.log" Jul 10 17:55:32.573602 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:32.573578 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n6672_6e9ce868-b07d-4674-be8b-3f27b781f174/dns-node-resolver/0.log" Jul 10 17:55:33.224825 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:33.224795 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-69wk5_2667319a-4b88-41ab-bba5-44f2b50e4269/node-ca/0.log" Jul 10 17:55:34.416870 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:34.416839 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545f68d885-h8rwn_d4db34a1-8704-4891-a9a6-5ee9e8108291/kube-auth-proxy/0.log" Jul 10 17:55:35.340954 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:35.340922 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k57hc_a75f6837-81ea-42a9-81c8-8808e1e8e44d/serve-healthcheck-canary/0.log" Jul 10 17:55:35.882531 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:35.882506 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-52fw9_caa0dfe9-a2b5-4812-869c-ba414370830f/kube-rbac-proxy/0.log" Jul 10 17:55:35.904736 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:35.904697 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-52fw9_caa0dfe9-a2b5-4812-869c-ba414370830f/exporter/0.log" Jul 10 17:55:35.927501 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:35.927471 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-52fw9_caa0dfe9-a2b5-4812-869c-ba414370830f/extractor/0.log" Jul 10 17:55:38.025140 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:38.025113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-jsx2z" Jul 10 17:55:38.777556 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:38.777521 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b884cfc77-d8l8z_fbead02a-0a11-42c8-a84d-d59c13dbb822/manager/0.log" Jul 10 17:55:38.811373 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:38.811348 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b884cfc77-d8l8z_fbead02a-0a11-42c8-a84d-d59c13dbb822/copy-manifests/0.log" Jul 10 17:55:40.321894 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:40.321857 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58d9f8fb8c-ph2dv_874b2bcb-91e0-4946-8e5c-5fc3b5a2c0dc/manager/0.log" Jul 10 17:55:41.107421 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:41.107354 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545f68d885-h8rwn_d4db34a1-8704-4891-a9a6-5ee9e8108291/kube-auth-proxy/0.log" Jul 10 17:55:42.712853 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:42.712822 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545f68d885-h8rwn_d4db34a1-8704-4891-a9a6-5ee9e8108291/kube-auth-proxy/0.log" Jul 10 17:55:47.203677 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.203647 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ngr2_671bb016-c27a-48ce-8aaa-408d5f05999f/kube-multus-additional-cni-plugins/0.log" Jul 10 17:55:47.223946 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.223912 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ngr2_671bb016-c27a-48ce-8aaa-408d5f05999f/egress-router-binary-copy/0.log" Jul 10 17:55:47.245822 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.245798 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ngr2_671bb016-c27a-48ce-8aaa-408d5f05999f/cni-plugins/0.log" Jul 10 17:55:47.269736 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.269715 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ngr2_671bb016-c27a-48ce-8aaa-408d5f05999f/bond-cni-plugin/0.log" Jul 10 17:55:47.291259 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.291238 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ngr2_671bb016-c27a-48ce-8aaa-408d5f05999f/routeoverride-cni/0.log" Jul 10 17:55:47.311557 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.311538 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ngr2_671bb016-c27a-48ce-8aaa-408d5f05999f/whereabouts-cni-bincopy/0.log" Jul 10 17:55:47.333274 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.333253 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ngr2_671bb016-c27a-48ce-8aaa-408d5f05999f/whereabouts-cni/0.log" Jul 10 17:55:47.565658 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.565580 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m5n45_4942f89d-529f-4889-82e4-0056933168ed/kube-multus/0.log" Jul 10 17:55:47.632847 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.632818 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cks5v_078c83e5-04a5-474a-8a60-4a0b8aa280d2/network-metrics-daemon/0.log" Jul 10 17:55:47.656833 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:47.656812 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cks5v_078c83e5-04a5-474a-8a60-4a0b8aa280d2/kube-rbac-proxy/0.log" Jul 10 17:55:48.539427 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:48.539402 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-controller/0.log" Jul 10 17:55:48.558351 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:48.558330 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/0.log" Jul 10 17:55:48.558351 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:48.558350 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovn-acl-logging/1.log" Jul 10 17:55:48.576917 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:48.576897 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/kube-rbac-proxy-node/0.log" Jul 10 17:55:48.600433 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:48.600412 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/kube-rbac-proxy-ovn-metrics/0.log" Jul 10 17:55:48.622318 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:48.622301 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/northd/0.log" Jul 10 17:55:48.645046 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:48.645020 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/nbdb/0.log" Jul 10 17:55:48.667071 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:48.667053 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/sbdb/0.log" Jul 10 17:55:48.714793 ip-10-0-140-169 kubenswrapper[2580]: I0710 17:55:48.714775 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jc2mr_ed52ac0a-561d-4c3f-b86c-b2ba3f1722b1/ovnkube-controller/0.log"