Jul 17 00:14:53.653186 ip-10-0-135-32 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 17 00:14:53.653197 ip-10-0-135-32 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 17 00:14:53.653204 ip-10-0-135-32 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 17 00:14:53.653381 ip-10-0-135-32 systemd[1]: Failed to start Kubernetes Kubelet. Jul 17 00:15:03.705659 ip-10-0-135-32 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 17 00:15:03.705677 ip-10-0-135-32 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ad030a05fa934ae984a1435e38abe822 -- Jul 17 00:17:05.561119 ip-10-0-135-32 systemd[1]: Starting Kubernetes Kubelet... Jul 17 00:17:07.571873 ip-10-0-135-32 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 00:17:07.571873 ip-10-0-135-32 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 00:17:07.571873 ip-10-0-135-32 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 00:17:07.571873 ip-10-0-135-32 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 00:17:07.571873 ip-10-0-135-32 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 00:17:07.571873 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.571128 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575749 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575766 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575770 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575773 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575776 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575779 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575781 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575784 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575787 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575790 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575792 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575795 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575797 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575801 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575806 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575809 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575811 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575816 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575821 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 00:17:07.576517 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575824 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575827 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575830 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575833 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575836 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575839 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575842 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575844 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575847 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575849 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575852 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575854 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575857 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575860 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575863 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575865 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575868 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575870 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575874 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 00:17:07.578260 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575878 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575903 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575907 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575910 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575913 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575915 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575918 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575922 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575924 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575927 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575930 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575934 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575937 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575941 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575945 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575948 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575951 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575954 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575957 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 00:17:07.579358 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575960 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575963 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575965 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575968 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575971 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575973 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575976 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575979 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575983 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575985 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575988 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575991 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575993 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575996 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.575999 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576002 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576004 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576008 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576011 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 00:17:07.580686 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576013 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576017 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576020 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576022 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576025 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576027 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576030 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576032 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576035 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576038 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576040 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576043 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576045 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576048 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576051 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576054 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576057 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576060 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576062 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576065 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 00:17:07.581559 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576067 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576070 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576073 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576841 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576849 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576852 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576855 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576858 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576862 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576866 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576868 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576871 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576875 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576879 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576899 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576904 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576907 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576910 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 00:17:07.582878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576914 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576917 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576921 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576924 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576926 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576929 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576932 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576934 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576937 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576940 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576942 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576944 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576947 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576950 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576952 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576955 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576958 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576960 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576963 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576966 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 00:17:07.583757 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576968 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576971 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576974 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576977 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576981 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576984 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576986 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576989 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576992 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576994 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.576997 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577000 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577003 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577006 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577008 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577011 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577013 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577016 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577019 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577021 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 00:17:07.585296 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577024 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577026 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577029 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577032 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577034 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577037 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577039 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577041 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577044 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577046 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577049 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577051 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577054 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577056 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577059 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577062 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577064 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577068 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577071 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577073 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 00:17:07.586166 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577076 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577078 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577081 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577083 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577086 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577089 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577092 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577094 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577098 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577102 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577105 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577108 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577112 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577115 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577118 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577121 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577123 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577126 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577129 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577132 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 00:17:07.588138 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577134 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577138 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577141 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577144 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577567 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577572 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577575 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577579 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577582 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577586 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577589 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577591 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577594 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577597 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577599 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577602 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577604 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577607 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577609 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577614 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 00:17:07.589040 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577617 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577621 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577624 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577626 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577629 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577632 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577634 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577637 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577640 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577642 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577645 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577647 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577650 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577652 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577655 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577659 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577661 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577664 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577666 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577670 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 00:17:07.590175 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577673 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577675 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577679 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577681 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577684 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577686 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577689 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577691 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577694 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577696 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577699 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577702 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577705 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577710 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577713 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577715 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577718 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577720 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577723 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577726 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 00:17:07.591072 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577728 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577731 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577733 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577736 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577739 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577741 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577744 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577746 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577749 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577751 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577755 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577758 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577761 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577764 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577766 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577769 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577772 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577774 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577777 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 00:17:07.592604 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577779 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577784 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577788 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577791 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577793 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577796 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577801 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577803 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577806 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577809 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577812 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577815 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577817 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577820 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577822 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577825 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577828 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577830 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577832 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 00:17:07.593492 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577835 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577837 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577840 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577843 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.577845 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577937 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577947 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577954 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577960 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577964 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577970 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577974 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577979 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577982 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577986 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577989 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577993 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577996 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.577999 2572 flags.go:64] FLAG: --cgroup-root="" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578004 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578007 2572 flags.go:64] FLAG: --client-ca-file="" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578010 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578013 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578017 2572 flags.go:64] FLAG: --cluster-domain="" Jul 17 00:17:07.594803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578020 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578023 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578026 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578030 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578034 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578037 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578040 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578044 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578047 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578050 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578055 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578058 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578061 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578065 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578068 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578072 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578075 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578078 2572 flags.go:64] FLAG: --enable-server="true" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578081 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578086 2572 flags.go:64] FLAG: --event-burst="100" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578089 2572 flags.go:64] FLAG: --event-qps="50" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578093 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578096 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578098 2572 flags.go:64] FLAG: --eviction-hard="" Jul 17 00:17:07.595907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578102 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578105 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578108 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578113 2572 flags.go:64] FLAG: --eviction-soft="" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578115 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578118 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578121 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578124 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578127 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578130 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578133 2572 flags.go:64] FLAG: --feature-gates="" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578136 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578139 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578143 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578146 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578149 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578153 2572 flags.go:64] FLAG: --help="false" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578156 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-135-32.ec2.internal" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578161 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578165 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578168 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578171 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578175 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578178 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 00:17:07.597678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578181 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578184 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578187 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578190 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578194 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578197 2572 flags.go:64] FLAG: --kube-reserved="" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578201 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578203 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578206 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578209 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578212 2572 flags.go:64] FLAG: --lock-file="" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578216 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578219 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578222 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578228 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578231 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578234 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578237 2572 flags.go:64] FLAG: --logging-format="text" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578240 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578243 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578246 2572 flags.go:64] FLAG: --manifest-url="" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578248 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578253 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578256 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578260 2572 flags.go:64] FLAG: --max-pods="110" Jul 17 00:17:07.598798 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578264 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578269 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578272 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578275 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578278 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578281 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578285 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578292 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578295 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578298 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578302 2572 flags.go:64] FLAG: --pod-cidr="" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578305 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578312 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578315 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578318 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578321 2572 flags.go:64] FLAG: --port="10250" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578324 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578327 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c48c28e14a2a66f5" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578330 2572 flags.go:64] FLAG: --qos-reserved="" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578334 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578337 2572 flags.go:64] FLAG: --register-node="true" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578340 2572 flags.go:64] FLAG: --register-with-taints="" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578344 2572 flags.go:64] FLAG: --registry-burst="10" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578347 2572 flags.go:64] FLAG: --registry-qps="5" Jul 17 00:17:07.600342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578349 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578352 2572 flags.go:64] FLAG: --reserved-memory="" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578356 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578359 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578362 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578365 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578368 2572 flags.go:64] FLAG: --runonce="false" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578371 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578374 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578377 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578382 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578385 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578388 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578391 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578394 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578397 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578400 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578403 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578406 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578409 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578412 2572 flags.go:64] FLAG: --system-cgroups="" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578419 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578425 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578427 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578430 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 00:17:07.601440 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578434 2572 flags.go:64] FLAG: --tls-min-version="" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578437 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578440 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578443 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578446 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578449 2572 flags.go:64] FLAG: --v="2" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578453 2572 flags.go:64] FLAG: --version="false" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578458 2572 flags.go:64] FLAG: --vmodule="" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578461 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578464 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578581 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578585 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578588 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578591 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578594 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578596 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578599 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578603 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578606 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578609 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578612 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578614 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 00:17:07.603183 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578617 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578619 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578622 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578625 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578627 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578630 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578632 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578636 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578638 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578641 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578643 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578646 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578649 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578652 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578654 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578656 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578659 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578662 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578664 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578667 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 00:17:07.665229 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578669 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578672 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578675 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578677 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578680 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578682 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578685 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578689 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578692 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578695 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578698 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578700 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578703 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578705 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578708 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578712 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578714 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578717 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578719 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578722 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 00:17:07.666465 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578725 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578727 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578730 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578735 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578738 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578741 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578744 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578746 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578749 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578752 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578754 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578756 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578759 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578761 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578764 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578766 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578769 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578771 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578774 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 00:17:07.667561 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578778 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578780 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578783 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578785 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578788 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578790 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578793 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578795 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578799 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578802 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578804 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578807 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578809 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578812 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578816 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578819 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578821 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578824 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578826 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578829 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 00:17:07.668452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578831 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578834 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578838 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578841 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578845 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578848 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578852 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.578855 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.578861 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.622452 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.622474 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.622496 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.622502 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622566 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 00:17:07.669531 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622571 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622575 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622579 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622582 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622586 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622590 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622594 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622597 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622600 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622603 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622606 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622609 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622612 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622614 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622617 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622620 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622623 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622626 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622629 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622631 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 00:17:07.670205 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622635 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622638 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622641 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622643 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622646 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622648 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622651 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622656 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622660 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622663 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622665 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622668 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622671 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622674 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622676 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622679 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622683 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622685 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622688 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 00:17:07.671314 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622691 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622694 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622697 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622700 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622702 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622705 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622708 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622711 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622713 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622716 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622719 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622721 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622724 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622727 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622730 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622732 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622735 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622738 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622741 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622744 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 00:17:07.672203 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622747 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622749 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622752 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622754 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622757 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622761 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622765 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622770 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622773 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622776 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622778 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622781 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622783 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622787 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622789 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622792 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622794 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622797 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622800 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 00:17:07.673118 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622803 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622806 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622808 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622811 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622813 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622816 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622819 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622822 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622825 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622827 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622830 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622832 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622835 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622837 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622840 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622842 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622845 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622847 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622850 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 00:17:07.676230 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.622853 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.622859 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623028 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623035 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623039 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623042 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623044 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623049 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623052 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623054 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623057 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623060 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623064 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623068 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623072 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 00:17:07.677110 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623075 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623078 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623080 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623084 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623087 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623090 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623092 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623095 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623097 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623100 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623103 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623105 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623108 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623111 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623114 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623116 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623119 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623122 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623124 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623127 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 00:17:07.678195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623129 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623132 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623134 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623137 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623139 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623142 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623144 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623147 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623150 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623153 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623155 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623158 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623162 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623164 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623167 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623170 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623172 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623174 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623177 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623179 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 00:17:07.679528 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623182 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623185 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623187 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623190 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623193 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623195 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623198 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623200 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623203 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623205 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623208 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623210 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623213 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623216 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623218 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623221 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623223 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623226 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623228 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623231 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 00:17:07.680436 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623233 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623236 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623238 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623241 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623243 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623247 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623249 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623253 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623257 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623260 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623263 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623266 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623268 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623271 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623274 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623277 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623279 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623283 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 00:17:07.681332 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623287 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623290 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623293 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623296 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623299 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623301 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623304 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:07.623311 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.623316 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.623460 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.625790 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.627133 2572 server.go:1019] "Starting client certificate rotation" Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.627233 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.627266 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 00:17:07.682445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.630122 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 00:17:07.682758 ip-10-0-135-32 systemd[1]: Started Kubernetes Kubelet. Jul 17 00:17:07.683211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.631476 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 00:17:07.683211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.640872 2572 log.go:25] "Validated CRI v1 runtime API" Jul 17 00:17:07.683211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.643981 2572 log.go:25] "Validated CRI v1 image API" Jul 17 00:17:07.683211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.645452 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 00:17:07.683211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.650953 2572 fs.go:135] Filesystem UUIDs: map[3392bc91-1490-45f0-bd2e-0db212679ffa:/dev/nvme0n1p4 65638362-6a2e-4e4e-b3c1-fd098f6f500c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 17 00:17:07.683211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.650971 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 00:17:07.683211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.658043 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.657286 2572 manager.go:216] Machine: {Timestamp:2026-07-17 00:17:07.655748043 +0000 UTC m=+0.402257068 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113860 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25e7e9168e53b0d78226eebe13670c SystemUUID:ec25e7e9-168e-53b0-d782-26eebe13670c BootID:ad030a05-fa93-4ae9-84a1-435e38abe822 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:81:94:c9:6e:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:81:94:c9:6e:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:ad:84:24:81:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.658095 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.658438 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.662377 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.662415 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-32.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.663838 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.663850 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.663858 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.664759 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.666091 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.669433 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.669569 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.669600 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.669613 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.669628 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.669637 2572 kubelet.go:398] "Adding apiserver pod source" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.669647 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 00:17:07.684089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.670054 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.670071 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.673130 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.674842 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.674879 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.675442 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n9pqj" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.675504 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.675518 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.675525 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.675531 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.675540 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.675545 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.676632 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.676640 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.676647 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.676653 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.676662 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.676669 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.678315 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.678325 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.678841 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.678967 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.680786 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n9pqj" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.681809 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-32.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.681845 2572 server.go:1268] "Started kubelet" Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.681950 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.682033 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.682054 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 00:17:07.685214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.684780 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 17 00:17:07.687576 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.686559 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 00:17:07.690729 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.689749 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 00:17:07.693532 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.693128 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.693753 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.693870 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694307 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694312 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694328 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694436 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694448 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694829 2572 factory.go:153] Registering CRI-O factory Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694842 2572 factory.go:223] Registration of the crio container factory successfully Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694906 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694916 2572 factory.go:55] Registering systemd factory Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694925 2572 factory.go:223] Registration of the systemd container factory successfully Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694948 2572 factory.go:103] Registering Raw factory Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.694962 2572 manager.go:1196] Started watching for new ooms in manager Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.695434 2572 manager.go:349] Starting recovery of all containers Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.696536 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.699915 2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-32.ec2.internal\" not found" node="ip-10-0-135-32.ec2.internal" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.707698 2572 manager.go:354] Recovery completed Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.711781 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.714195 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.714227 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.714242 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.714685 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.714696 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.714701 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.714716 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.715527 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.715558 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.715572 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.716490 2572 policy_none.go:49] "None policy: Start" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.716505 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.716516 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 00:17:07.765106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.717806 2572 policy_none.go:47] "Start" Jul 17 00:17:07.840425 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.794588 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.894850 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.898645 2572 manager.go:337] "Starting Device Plugin manager" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.898690 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.898700 2572 server.go:87] "Starting device plugin registration server" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.899022 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.899037 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.899221 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.899320 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:07.899329 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.899940 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 00:17:07.941260 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:07.899992 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.025973 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.027165 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.027194 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.027215 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.027242 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.045985 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.046012 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-32.ec2.internal\": node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.076579 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.079521 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.080949 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.080990 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.081015 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.081066 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 00:17:08.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.083901 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 00:17:08.177477 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.177406 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.181581 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.181556 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal"] Jul 17 00:17:08.181676 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.181661 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 00:17:08.182759 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.182742 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Jul 17 00:17:08.182855 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.182778 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 00:17:08.182855 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.182793 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Jul 17 00:17:08.184062 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.184046 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 00:17:08.184241 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.184226 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.184241 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.184252 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 00:17:08.184857 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.184837 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Jul 17 00:17:08.184989 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.184870 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 00:17:08.184989 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.184899 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Jul 17 00:17:08.184989 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.184870 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Jul 17 00:17:08.184989 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.184944 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 00:17:08.184989 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.184955 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Jul 17 00:17:08.186355 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.186339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.186442 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.186362 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 00:17:08.187445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.187427 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Jul 17 00:17:08.187537 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.187453 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 00:17:08.187537 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.187465 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Jul 17 00:17:08.201485 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.201461 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-32.ec2.internal\" not found" node="ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.205442 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.205426 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-32.ec2.internal\" not found" node="ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.228107 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.228080 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29420c943b7433d6268f022fb62a1398-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"29420c943b7433d6268f022fb62a1398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.228216 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.228110 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c41058aa23e327c16b34dda6898223f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-32.ec2.internal\" (UID: \"9c41058aa23e327c16b34dda6898223f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.228216 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.228136 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29420c943b7433d6268f022fb62a1398-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"29420c943b7433d6268f022fb62a1398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.277712 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.277676 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.328684 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.328646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c41058aa23e327c16b34dda6898223f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-32.ec2.internal\" (UID: \"9c41058aa23e327c16b34dda6898223f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.328792 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.328689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29420c943b7433d6268f022fb62a1398-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"29420c943b7433d6268f022fb62a1398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.328792 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.328720 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c41058aa23e327c16b34dda6898223f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-32.ec2.internal\" (UID: \"9c41058aa23e327c16b34dda6898223f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.328792 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.328744 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29420c943b7433d6268f022fb62a1398-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"29420c943b7433d6268f022fb62a1398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.328792 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.328782 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29420c943b7433d6268f022fb62a1398-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"29420c943b7433d6268f022fb62a1398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.328965 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.328790 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29420c943b7433d6268f022fb62a1398-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"29420c943b7433d6268f022fb62a1398\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.378809 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.378761 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.479307 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.479246 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.503596 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.503569 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.509767 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.509743 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Jul 17 00:17:08.579946 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.579906 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.626235 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.626188 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 00:17:08.626417 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.626397 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 00:17:08.626476 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.626417 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 00:17:08.680709 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.680674 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.685493 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.685466 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 00:12:07 +0000 UTC" deadline="2028-03-13 07:45:01.179227263 +0000 UTC" Jul 17 00:17:08.685543 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:08.685493 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14527h27m52.493737205s" Jul 17 00:17:08.781654 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.781562 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.882209 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.882172 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:08.982870 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:08.982840 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:09.082977 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.082881 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:09.134135 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.134105 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 00:17:09.183328 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.183294 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Jul 17 00:17:09.259015 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.258993 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 00:17:09.294994 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.294958 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Jul 17 00:17:09.306456 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.306433 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 00:17:09.307312 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.307298 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Jul 17 00:17:09.312741 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.312726 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 00:17:09.562784 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.562693 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 00:17:09.670649 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.670599 2572 apiserver.go:52] "Watching apiserver" Jul 17 00:17:09.678157 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.678128 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 00:17:09.681258 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.681227 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-n78sr","kube-system/konnectivity-agent-m676v","kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq","openshift-dns/node-resolver-9tj4h","openshift-image-registry/node-ca-mf9bv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal","openshift-multus/multus-additional-cni-plugins-8fbm4","openshift-cluster-node-tuning-operator/tuned-dvkxv","openshift-multus/multus-cx5p6","openshift-multus/network-metrics-daemon-75rcn","openshift-network-diagnostics/network-check-target-xvfn6","openshift-network-operator/iptables-alerter-qk4c4","openshift-ovn-kubernetes/ovnkube-node-9qbd7"] Jul 17 00:17:09.683138 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.683116 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.684829 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.684806 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.685482 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.685467 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 00:17:09.685551 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.685512 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 00:17:09.685731 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.685719 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 00:17:09.685793 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.685780 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 00:17:09.686028 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.686012 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jfn9h\"" Jul 17 00:17:09.686102 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.686063 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.687271 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.687236 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 00:17:09.687365 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.687352 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.687870 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.687842 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 00:17:09.688004 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.687905 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 00:17:09.688004 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.687960 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zx5c2\"" Jul 17 00:17:09.688711 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.688695 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.689312 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689292 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 00:17:09.689424 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689315 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 00:17:09.689424 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689324 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 00:17:09.689424 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689331 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5td26\"" Jul 17 00:17:09.689424 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689326 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 00:17:09.689424 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689420 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 00:17:09.689645 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689421 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 00:17:09.689645 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689572 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 00:17:09.689645 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689554 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 00:17:09.689751 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689595 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4cmnz\"" Jul 17 00:17:09.689995 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.689977 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 00:17:09.690177 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.690165 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.690765 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.690748 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 00:17:09.690822 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.690811 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8v6vs\"" Jul 17 00:17:09.691078 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.691066 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 00:17:09.691426 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.691411 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.692315 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.692299 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 00:17:09.692413 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.692399 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 00:17:09.692504 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.692435 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sqd24\"" Jul 17 00:17:09.692684 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.692671 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 00:17:09.693341 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.693326 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 00:17:09.693428 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.693399 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 00:17:09.693481 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.693328 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.693550 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.693506 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dj9sv\"" Jul 17 00:17:09.693921 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.693908 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 00:17:09.695207 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.695192 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:09.695552 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.695533 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 00:17:09.695621 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.695577 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pbctf\"" Jul 17 00:17:09.695621 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.695582 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 00:17:09.697874 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.696289 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:09.697874 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.696370 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:09.697874 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.697425 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 00:17:09.697874 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.697672 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 00:17:09.697874 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.697759 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g8c9m\"" Jul 17 00:17:09.698860 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.698838 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:09.698959 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.698936 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:09.700552 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.700534 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:09.700643 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.700593 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:09.705787 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.705769 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 00:17:09.728317 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.728284 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxlv5" Jul 17 00:17:09.733762 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.733742 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxlv5" Jul 17 00:17:09.734832 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.734809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-run-ovn\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.734936 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.734845 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-cni-bin\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.734936 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.734870 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16fd2ba4-d022-4232-938c-ee9ab79b44f9-ovn-node-metrics-cert\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.734936 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.734909 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ckzz\" (UniqueName: \"kubernetes.io/projected/16fd2ba4-d022-4232-938c-ee9ab79b44f9-kube-api-access-7ckzz\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.734936 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.734931 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-systemd\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.735133 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.734957 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2sk5\" (UniqueName: \"kubernetes.io/projected/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-kube-api-access-f2sk5\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.735133 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.734982 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-cnibin\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.735133 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735016 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16fd2ba4-d022-4232-938c-ee9ab79b44f9-env-overrides\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.735133 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bd7118e-95e9-46b8-9e06-2579ca5607c3-konnectivity-ca\") pod \"konnectivity-agent-m676v\" (UID: \"1bd7118e-95e9-46b8-9e06-2579ca5607c3\") " pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:09.735133 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bfe148b-94d1-4c6b-9edf-aa448a273ba1-host-slash\") pod \"iptables-alerter-qk4c4\" (UID: \"2bfe148b-94d1-4c6b-9edf-aa448a273ba1\") " pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.735133 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735087 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-sys-fs\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.735133 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735109 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-run-netns\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.735363 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735135 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-slash\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.735363 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735158 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-run-systemd\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.735363 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735214 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l277g\" (UniqueName: \"kubernetes.io/projected/2bfe148b-94d1-4c6b-9edf-aa448a273ba1-kube-api-access-l277g\") pod \"iptables-alerter-qk4c4\" (UID: \"2bfe148b-94d1-4c6b-9edf-aa448a273ba1\") " pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.735363 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735251 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b254ab0-677e-4fd2-b0a2-ea3e76928696-cni-binary-copy\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.735363 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735268 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-cni-netd\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.735363 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735286 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1faae122-db93-4c37-bf65-16ed121ad6dc-host\") pod \"node-ca-mf9bv\" (UID: \"1faae122-db93-4c37-bf65-16ed121ad6dc\") " pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.735363 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735316 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-multus-cni-dir\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.735363 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735339 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.735363 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735369 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16fd2ba4-d022-4232-938c-ee9ab79b44f9-ovnkube-script-lib\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735383 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bfe148b-94d1-4c6b-9edf-aa448a273ba1-iptables-alerter-script\") pod \"iptables-alerter-qk4c4\" (UID: \"2bfe148b-94d1-4c6b-9edf-aa448a273ba1\") " pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735410 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-run\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735453 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-os-release\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80c093d7-478e-4edf-820c-34720b639a43-cni-binary-copy\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735518 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b254ab0-677e-4fd2-b0a2-ea3e76928696-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-kubernetes\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735564 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16fd2ba4-d022-4232-938c-ee9ab79b44f9-ovnkube-config\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735585 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180b2640-39c5-4cf3-98c6-62deb22ebd32-tmp-dir\") pod \"node-resolver-9tj4h\" (UID: \"180b2640-39c5-4cf3-98c6-62deb22ebd32\") " pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735601 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-cnibin\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-multus-socket-dir-parent\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-run-k8s-cni-cncf-io\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735655 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1faae122-db93-4c37-bf65-16ed121ad6dc-serviceca\") pod \"node-ca-mf9bv\" (UID: \"1faae122-db93-4c37-bf65-16ed121ad6dc\") " pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735671 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-run-multus-certs\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735686 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-run-netns\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.735706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735702 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735716 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxxk8\" (UniqueName: \"kubernetes.io/projected/6b254ab0-677e-4fd2-b0a2-ea3e76928696-kube-api-access-xxxk8\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735732 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735763 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735796 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-device-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735832 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-tuned\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735855 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-tmp\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735877 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-system-cni-dir\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735913 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh4br\" (UniqueName: \"kubernetes.io/projected/80c093d7-478e-4edf-820c-34720b639a43-kube-api-access-hh4br\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735936 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-etc-openvswitch\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735959 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/180b2640-39c5-4cf3-98c6-62deb22ebd32-hosts-file\") pod \"node-resolver-9tj4h\" (UID: \"180b2640-39c5-4cf3-98c6-62deb22ebd32\") " pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.735993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-os-release\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736018 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-run-openvswitch\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-lib-modules\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736063 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b254ab0-677e-4fd2-b0a2-ea3e76928696-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736092 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-kubelet\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.736402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736115 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bd7118e-95e9-46b8-9e06-2579ca5607c3-agent-certs\") pod \"konnectivity-agent-m676v\" (UID: \"1bd7118e-95e9-46b8-9e06-2579ca5607c3\") " pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736138 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlz9\" (UniqueName: \"kubernetes.io/projected/180b2640-39c5-4cf3-98c6-62deb22ebd32-kube-api-access-2mlz9\") pod \"node-resolver-9tj4h\" (UID: \"180b2640-39c5-4cf3-98c6-62deb22ebd32\") " pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlg4g\" (UniqueName: \"kubernetes.io/projected/1faae122-db93-4c37-bf65-16ed121ad6dc-kube-api-access-tlg4g\") pod \"node-ca-mf9bv\" (UID: \"1faae122-db93-4c37-bf65-16ed121ad6dc\") " pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-sysconfig\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736220 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-socket-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736237 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-registration-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736272 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2d4d\" (UniqueName: \"kubernetes.io/projected/16326435-7285-4778-a104-38a9df454e3c-kube-api-access-h2d4d\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-sysctl-conf\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-var-lib-kubelet\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736373 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-var-lib-openvswitch\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736395 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-node-log\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-system-cni-dir\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-var-lib-kubelet\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736467 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-host\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736483 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-modprobe-d\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736506 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-sys\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.736880 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736522 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-var-lib-cni-bin\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.737362 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-multus-conf-dir\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.737362 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736562 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80c093d7-478e-4edf-820c-34720b639a43-multus-daemon-config\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.737362 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-etc-kubernetes\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.737362 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736595 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-systemd-units\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.737362 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736609 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-log-socket\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.737362 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736634 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-sysctl-d\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.737362 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736661 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-var-lib-cni-multus\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.737362 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.736685 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-hostroot\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.795108 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.795083 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 00:17:09.837694 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837667 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.837777 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837714 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxxk8\" (UniqueName: \"kubernetes.io/projected/6b254ab0-677e-4fd2-b0a2-ea3e76928696-kube-api-access-xxxk8\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.837829 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.837829 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837803 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.837946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-device-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.837946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837864 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-tuned\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.837946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837900 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.837946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-tmp\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.837946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837931 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.838161 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837948 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.838161 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-system-cni-dir\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.838161 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.837989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh4br\" (UniqueName: \"kubernetes.io/projected/80c093d7-478e-4edf-820c-34720b639a43-kube-api-access-hh4br\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.838161 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-etc-openvswitch\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.838161 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838051 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-system-cni-dir\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.838161 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/180b2640-39c5-4cf3-98c6-62deb22ebd32-hosts-file\") pod \"node-resolver-9tj4h\" (UID: \"180b2640-39c5-4cf3-98c6-62deb22ebd32\") " pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.838161 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838013 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-device-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.838161 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838105 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-etc-openvswitch\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.838161 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838134 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282hg\" (UniqueName: \"kubernetes.io/projected/dc74d918-a560-4e0b-831f-c09d6a1826e9-kube-api-access-282hg\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838207 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/180b2640-39c5-4cf3-98c6-62deb22ebd32-hosts-file\") pod \"node-resolver-9tj4h\" (UID: \"180b2640-39c5-4cf3-98c6-62deb22ebd32\") " pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838226 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838239 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-os-release\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-run-openvswitch\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838310 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838349 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-run-openvswitch\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838346 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-os-release\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838368 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-lib-modules\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b254ab0-677e-4fd2-b0a2-ea3e76928696-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-kubelet\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.838527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838484 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-lib-modules\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.838946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838536 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bd7118e-95e9-46b8-9e06-2579ca5607c3-agent-certs\") pod \"konnectivity-agent-m676v\" (UID: \"1bd7118e-95e9-46b8-9e06-2579ca5607c3\") " pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:09.838946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838561 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlz9\" (UniqueName: \"kubernetes.io/projected/180b2640-39c5-4cf3-98c6-62deb22ebd32-kube-api-access-2mlz9\") pod \"node-resolver-9tj4h\" (UID: \"180b2640-39c5-4cf3-98c6-62deb22ebd32\") " pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.838946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838586 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlg4g\" (UniqueName: \"kubernetes.io/projected/1faae122-db93-4c37-bf65-16ed121ad6dc-kube-api-access-tlg4g\") pod \"node-ca-mf9bv\" (UID: \"1faae122-db93-4c37-bf65-16ed121ad6dc\") " pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.838946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838602 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-kubelet\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.838946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838717 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-sysconfig\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.838946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.838748 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-socket-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.839433 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.839378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-registration-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.839560 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.839453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2d4d\" (UniqueName: \"kubernetes.io/projected/16326435-7285-4778-a104-38a9df454e3c-kube-api-access-h2d4d\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.839560 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.839484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-sysctl-conf\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.839688 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.839672 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-sysconfig\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.839776 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.839736 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-registration-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.839776 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.839766 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-sysctl-conf\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840020 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840002 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-socket-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.840133 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-var-lib-kubelet\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.840194 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-var-lib-openvswitch\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.840194 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840163 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-var-lib-kubelet\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.840289 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840193 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-node-log\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.840289 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840228 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-system-cni-dir\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.840289 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840235 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-var-lib-openvswitch\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.840289 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840262 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-var-lib-kubelet\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-host\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840299 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-node-log\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.840505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840372 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-system-cni-dir\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.840505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840398 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-var-lib-kubelet\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-modprobe-d\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840445 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-sys\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840477 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-var-lib-cni-bin\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.840505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840488 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-modprobe-d\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840507 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-multus-conf-dir\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.840937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840538 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80c093d7-478e-4edf-820c-34720b639a43-multus-daemon-config\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.840937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840562 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-sys\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-etc-kubernetes\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.840937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840623 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-systemd-units\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.840937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840638 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-host\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840667 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-log-socket\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.840937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-sysctl-d\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.840937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840802 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-etc-kubernetes\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.840937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840842 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-var-lib-cni-multus\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840976 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-sysctl-d\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.840986 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-var-lib-cni-multus\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841032 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-var-lib-cni-bin\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841049 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-systemd-units\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-hostroot\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841109 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-multus-conf-dir\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841109 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-run-ovn\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-cni-bin\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841164 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-log-socket\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16fd2ba4-d022-4232-938c-ee9ab79b44f9-ovn-node-metrics-cert\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841214 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-run-ovn\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-hostroot\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.841377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841309 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-cni-bin\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841203 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ckzz\" (UniqueName: \"kubernetes.io/projected/16fd2ba4-d022-4232-938c-ee9ab79b44f9-kube-api-access-7ckzz\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841688 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-kubelet-config\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-systemd\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2sk5\" (UniqueName: \"kubernetes.io/projected/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-kube-api-access-f2sk5\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841798 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-cnibin\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16fd2ba4-d022-4232-938c-ee9ab79b44f9-env-overrides\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bd7118e-95e9-46b8-9e06-2579ca5607c3-konnectivity-ca\") pod \"konnectivity-agent-m676v\" (UID: \"1bd7118e-95e9-46b8-9e06-2579ca5607c3\") " pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bfe148b-94d1-4c6b-9edf-aa448a273ba1-host-slash\") pod \"iptables-alerter-qk4c4\" (UID: \"2bfe148b-94d1-4c6b-9edf-aa448a273ba1\") " pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-sys-fs\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.841971 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:09.841998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-run-netns\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842031 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-slash\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842067 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-tmp\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842071 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-tuned\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-run-systemd\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l277g\" (UniqueName: \"kubernetes.io/projected/2bfe148b-94d1-4c6b-9edf-aa448a273ba1-kube-api-access-l277g\") pod \"iptables-alerter-qk4c4\" (UID: \"2bfe148b-94d1-4c6b-9edf-aa448a273ba1\") " pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-run-systemd\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842333 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bfe148b-94d1-4c6b-9edf-aa448a273ba1-host-slash\") pod \"iptables-alerter-qk4c4\" (UID: \"2bfe148b-94d1-4c6b-9edf-aa448a273ba1\") " pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842403 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-systemd\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-sys-fs\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842447 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842496 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b254ab0-677e-4fd2-b0a2-ea3e76928696-cni-binary-copy\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.842639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842542 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-cni-netd\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.843256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.842817 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bd7118e-95e9-46b8-9e06-2579ca5607c3-konnectivity-ca\") pod \"konnectivity-agent-m676v\" (UID: \"1bd7118e-95e9-46b8-9e06-2579ca5607c3\") " pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:09.843256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843094 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1faae122-db93-4c37-bf65-16ed121ad6dc-host\") pod \"node-ca-mf9bv\" (UID: \"1faae122-db93-4c37-bf65-16ed121ad6dc\") " pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.843256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-multus-cni-dir\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.843256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843179 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.843256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843185 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-slash\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.843256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843225 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.843516 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-cnibin\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843793 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16fd2ba4-d022-4232-938c-ee9ab79b44f9-ovnkube-script-lib\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843847 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-cni-netd\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bfe148b-94d1-4c6b-9edf-aa448a273ba1-iptables-alerter-script\") pod \"iptables-alerter-qk4c4\" (UID: \"2bfe148b-94d1-4c6b-9edf-aa448a273ba1\") " pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843957 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-run\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844014 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-os-release\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844130 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16fd2ba4-d022-4232-938c-ee9ab79b44f9-ovn-node-metrics-cert\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844182 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80c093d7-478e-4edf-820c-34720b639a43-cni-binary-copy\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.843808 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844253 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bd7118e-95e9-46b8-9e06-2579ca5607c3-agent-certs\") pod \"konnectivity-agent-m676v\" (UID: \"1bd7118e-95e9-46b8-9e06-2579ca5607c3\") " pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844260 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b254ab0-677e-4fd2-b0a2-ea3e76928696-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844311 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-kubernetes\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844324 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16fd2ba4-d022-4232-938c-ee9ab79b44f9-env-overrides\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844344 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1faae122-db93-4c37-bf65-16ed121ad6dc-host\") pod \"node-ca-mf9bv\" (UID: \"1faae122-db93-4c37-bf65-16ed121ad6dc\") " pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16fd2ba4-d022-4232-938c-ee9ab79b44f9-ovnkube-config\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844400 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16326435-7285-4778-a104-38a9df454e3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180b2640-39c5-4cf3-98c6-62deb22ebd32-tmp-dir\") pod \"node-resolver-9tj4h\" (UID: \"180b2640-39c5-4cf3-98c6-62deb22ebd32\") " pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844503 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-multus-cni-dir\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.845846 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844496 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-cnibin\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844553 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-multus-socket-dir-parent\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844582 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-run-k8s-cni-cncf-io\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844668 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1faae122-db93-4c37-bf65-16ed121ad6dc-serviceca\") pod \"node-ca-mf9bv\" (UID: \"1faae122-db93-4c37-bf65-16ed121ad6dc\") " pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844720 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-dbus\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844774 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-run-multus-certs\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.844862 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bfe148b-94d1-4c6b-9edf-aa448a273ba1-iptables-alerter-script\") pod \"iptables-alerter-qk4c4\" (UID: \"2bfe148b-94d1-4c6b-9edf-aa448a273ba1\") " pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845084 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-run-netns\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845175 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80c093d7-478e-4edf-820c-34720b639a43-cni-binary-copy\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845200 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-os-release\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845236 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16fd2ba4-d022-4232-938c-ee9ab79b44f9-host-run-netns\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-run-netns\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845277 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-etc-kubernetes\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845283 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-run-multus-certs\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845316 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-multus-socket-dir-parent\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845318 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80c093d7-478e-4edf-820c-34720b639a43-host-run-k8s-cni-cncf-io\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845336 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b254ab0-677e-4fd2-b0a2-ea3e76928696-cni-binary-copy\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845360 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-run\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.846951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845489 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b254ab0-677e-4fd2-b0a2-ea3e76928696-cnibin\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.847755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845788 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1faae122-db93-4c37-bf65-16ed121ad6dc-serviceca\") pod \"node-ca-mf9bv\" (UID: \"1faae122-db93-4c37-bf65-16ed121ad6dc\") " pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.847755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845807 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180b2640-39c5-4cf3-98c6-62deb22ebd32-tmp-dir\") pod \"node-resolver-9tj4h\" (UID: \"180b2640-39c5-4cf3-98c6-62deb22ebd32\") " pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.847755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845851 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16fd2ba4-d022-4232-938c-ee9ab79b44f9-ovnkube-config\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.847755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.845868 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b254ab0-677e-4fd2-b0a2-ea3e76928696-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.847755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.846096 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80c093d7-478e-4edf-820c-34720b639a43-multus-daemon-config\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.847755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.846326 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b254ab0-677e-4fd2-b0a2-ea3e76928696-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.847755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.846750 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16fd2ba4-d022-4232-938c-ee9ab79b44f9-ovnkube-script-lib\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.849356 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.849328 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2d4d\" (UniqueName: \"kubernetes.io/projected/16326435-7285-4778-a104-38a9df454e3c-kube-api-access-h2d4d\") pod \"aws-ebs-csi-driver-node-xxjxq\" (UID: \"16326435-7285-4778-a104-38a9df454e3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:09.849448 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.849389 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlg4g\" (UniqueName: \"kubernetes.io/projected/1faae122-db93-4c37-bf65-16ed121ad6dc-kube-api-access-tlg4g\") pod \"node-ca-mf9bv\" (UID: \"1faae122-db93-4c37-bf65-16ed121ad6dc\") " pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:09.849448 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.849424 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxxk8\" (UniqueName: \"kubernetes.io/projected/6b254ab0-677e-4fd2-b0a2-ea3e76928696-kube-api-access-xxxk8\") pod \"multus-additional-cni-plugins-8fbm4\" (UID: \"6b254ab0-677e-4fd2-b0a2-ea3e76928696\") " pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:09.849738 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.849718 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlz9\" (UniqueName: \"kubernetes.io/projected/180b2640-39c5-4cf3-98c6-62deb22ebd32-kube-api-access-2mlz9\") pod \"node-resolver-9tj4h\" (UID: \"180b2640-39c5-4cf3-98c6-62deb22ebd32\") " pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:09.850359 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.850341 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh4br\" (UniqueName: \"kubernetes.io/projected/80c093d7-478e-4edf-820c-34720b639a43-kube-api-access-hh4br\") pod \"multus-cx5p6\" (UID: \"80c093d7-478e-4edf-820c-34720b639a43\") " pod="openshift-multus/multus-cx5p6" Jul 17 00:17:09.851265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.851242 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2sk5\" (UniqueName: \"kubernetes.io/projected/1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6-kube-api-access-f2sk5\") pod \"tuned-dvkxv\" (UID: \"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6\") " pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:09.851647 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.851626 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ckzz\" (UniqueName: \"kubernetes.io/projected/16fd2ba4-d022-4232-938c-ee9ab79b44f9-kube-api-access-7ckzz\") pod \"ovnkube-node-9qbd7\" (UID: \"16fd2ba4-d022-4232-938c-ee9ab79b44f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:09.851683 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.851630 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l277g\" (UniqueName: \"kubernetes.io/projected/2bfe148b-94d1-4c6b-9edf-aa448a273ba1-kube-api-access-l277g\") pod \"iptables-alerter-qk4c4\" (UID: \"2bfe148b-94d1-4c6b-9edf-aa448a273ba1\") " pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:09.945860 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.945813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-kubelet-config\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:09.946082 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.945902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:09.946082 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.945954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:09.946082 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.945988 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-kubelet-config\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:09.946082 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.946054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-dbus\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:09.946308 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.946095 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:09.946308 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.946141 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282hg\" (UniqueName: \"kubernetes.io/projected/dc74d918-a560-4e0b-831f-c09d6a1826e9-kube-api-access-282hg\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:09.946308 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.946178 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:09.946308 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.946198 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret podName:dfabc064-bfdc-4cb0-923c-eb32d6ab392b nodeName:}" failed. No retries permitted until 2026-07-17 00:17:10.446172199 +0000 UTC m=+3.192681235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret") pod "global-pull-secret-syncer-n78sr" (UID: "dfabc064-bfdc-4cb0-923c-eb32d6ab392b") : object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:09.946308 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.946304 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:09.946587 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.946355 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs podName:dc74d918-a560-4e0b-831f-c09d6a1826e9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:10.446339024 +0000 UTC m=+3.192848054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs") pod "network-metrics-daemon-75rcn" (UID: "dc74d918-a560-4e0b-831f-c09d6a1826e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:09.946587 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.946410 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-dbus\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:09.952635 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.952607 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 00:17:09.952635 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.952639 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 00:17:09.952829 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.952650 2572 projected.go:196] Error preparing data for projected volume kube-api-access-bkkcw for pod openshift-network-diagnostics/network-check-target-xvfn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:09.952829 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:09.952728 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw podName:419c05e1-c598-4835-b870-69336fd5dea9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:10.452710852 +0000 UTC m=+3.199219877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bkkcw" (UniqueName: "kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw") pod "network-check-target-xvfn6" (UID: "419c05e1-c598-4835-b870-69336fd5dea9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:09.954995 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:09.954967 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-282hg\" (UniqueName: \"kubernetes.io/projected/dc74d918-a560-4e0b-831f-c09d6a1826e9-kube-api-access-282hg\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:10.013733 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.013702 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cx5p6" Jul 17 00:17:10.031871 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.031821 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qk4c4" Jul 17 00:17:10.038808 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.038783 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:10.065043 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.065009 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c41058aa23e327c16b34dda6898223f.slice/crio-3b115e3717c679d6f7f9a65c0735b526b95c6895d5f7749d91772e529054bcf6 WatchSource:0}: Error finding container 3b115e3717c679d6f7f9a65c0735b526b95c6895d5f7749d91772e529054bcf6: Status 404 returned error can't find the container with id 3b115e3717c679d6f7f9a65c0735b526b95c6895d5f7749d91772e529054bcf6 Jul 17 00:17:10.065590 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.065569 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16fd2ba4_d022_4232_938c_ee9ab79b44f9.slice/crio-372a646aaa0a154c80be8a63e47de84a61f734213d140eef874dfcca2a7cac86 WatchSource:0}: Error finding container 372a646aaa0a154c80be8a63e47de84a61f734213d140eef874dfcca2a7cac86: Status 404 returned error can't find the container with id 372a646aaa0a154c80be8a63e47de84a61f734213d140eef874dfcca2a7cac86 Jul 17 00:17:10.066195 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.066130 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29420c943b7433d6268f022fb62a1398.slice/crio-327174e0324f61c70bb1a2854df10b0ab66f9fb0c19d6c0ad6019fedda69895c WatchSource:0}: Error finding container 327174e0324f61c70bb1a2854df10b0ab66f9fb0c19d6c0ad6019fedda69895c: Status 404 returned error can't find the container with id 327174e0324f61c70bb1a2854df10b0ab66f9fb0c19d6c0ad6019fedda69895c Jul 17 00:17:10.066688 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.066658 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" Jul 17 00:17:10.066878 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.066861 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfe148b_94d1_4c6b_9edf_aa448a273ba1.slice/crio-ee6916e2d68bfd33a0e1cc4244abed76ada580f23169aeefd2aae6566df69843 WatchSource:0}: Error finding container ee6916e2d68bfd33a0e1cc4244abed76ada580f23169aeefd2aae6566df69843: Status 404 returned error can't find the container with id ee6916e2d68bfd33a0e1cc4244abed76ada580f23169aeefd2aae6566df69843 Jul 17 00:17:10.073750 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.073734 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 00:17:10.078215 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.078195 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16326435_7285_4778_a104_38a9df454e3c.slice/crio-d944ee2ebebc66e92939ab61b9d3eebce612f0e299e04933114398e87c88eee4 WatchSource:0}: Error finding container d944ee2ebebc66e92939ab61b9d3eebce612f0e299e04933114398e87c88eee4: Status 404 returned error can't find the container with id d944ee2ebebc66e92939ab61b9d3eebce612f0e299e04933114398e87c88eee4 Jul 17 00:17:10.081555 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.081537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9tj4h" Jul 17 00:17:10.085514 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.085406 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" event={"ID":"16fd2ba4-d022-4232-938c-ee9ab79b44f9","Type":"ContainerStarted","Data":"372a646aaa0a154c80be8a63e47de84a61f734213d140eef874dfcca2a7cac86"} Jul 17 00:17:10.086769 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.086746 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" event={"ID":"16326435-7285-4778-a104-38a9df454e3c","Type":"ContainerStarted","Data":"d944ee2ebebc66e92939ab61b9d3eebce612f0e299e04933114398e87c88eee4"} Jul 17 00:17:10.088346 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.088322 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qk4c4" event={"ID":"2bfe148b-94d1-4c6b-9edf-aa448a273ba1","Type":"ContainerStarted","Data":"ee6916e2d68bfd33a0e1cc4244abed76ada580f23169aeefd2aae6566df69843"} Jul 17 00:17:10.089586 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.089559 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" event={"ID":"29420c943b7433d6268f022fb62a1398","Type":"ContainerStarted","Data":"327174e0324f61c70bb1a2854df10b0ab66f9fb0c19d6c0ad6019fedda69895c"} Jul 17 00:17:10.090035 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.090013 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180b2640_39c5_4cf3_98c6_62deb22ebd32.slice/crio-4f73720a4121b6edee0d9aea6f3fb0daef4377c0f3263461898100e088cecb77 WatchSource:0}: Error finding container 4f73720a4121b6edee0d9aea6f3fb0daef4377c0f3263461898100e088cecb77: Status 404 returned error can't find the container with id 4f73720a4121b6edee0d9aea6f3fb0daef4377c0f3263461898100e088cecb77 Jul 17 00:17:10.090679 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.090645 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" event={"ID":"9c41058aa23e327c16b34dda6898223f","Type":"ContainerStarted","Data":"3b115e3717c679d6f7f9a65c0735b526b95c6895d5f7749d91772e529054bcf6"} Jul 17 00:17:10.098676 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.098657 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mf9bv" Jul 17 00:17:10.105619 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.105601 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faae122_db93_4c37_bf65_16ed121ad6dc.slice/crio-3fc38393f670c34b7c03d5bbabaa3a99cb5047ad29206e7dad5487e5ede5a3c1 WatchSource:0}: Error finding container 3fc38393f670c34b7c03d5bbabaa3a99cb5047ad29206e7dad5487e5ede5a3c1: Status 404 returned error can't find the container with id 3fc38393f670c34b7c03d5bbabaa3a99cb5047ad29206e7dad5487e5ede5a3c1 Jul 17 00:17:10.112930 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.112910 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" Jul 17 00:17:10.120298 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.120271 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcc2c34_a1aa_4286_a4d8_c3b1b1fb8af6.slice/crio-11fe2f25b45ad367b6f76151153481f0c1f86248b970b262db7ea3df8e2f7494 WatchSource:0}: Error finding container 11fe2f25b45ad367b6f76151153481f0c1f86248b970b262db7ea3df8e2f7494: Status 404 returned error can't find the container with id 11fe2f25b45ad367b6f76151153481f0c1f86248b970b262db7ea3df8e2f7494 Jul 17 00:17:10.133752 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.133725 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" Jul 17 00:17:10.139011 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.138988 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:10.141619 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.141598 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b254ab0_677e_4fd2_b0a2_ea3e76928696.slice/crio-f17066572fff0d298c38598e2d146583ce1169259eddbb9f90ff04c013dad570 WatchSource:0}: Error finding container f17066572fff0d298c38598e2d146583ce1169259eddbb9f90ff04c013dad570: Status 404 returned error can't find the container with id f17066572fff0d298c38598e2d146583ce1169259eddbb9f90ff04c013dad570 Jul 17 00:17:10.146320 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.146299 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd7118e_95e9_46b8_9e06_2579ca5607c3.slice/crio-09e1ea3686db34e3b8001128dbdd5b512aeb9d9a89963aff70b0b5d49e02c5f4 WatchSource:0}: Error finding container 09e1ea3686db34e3b8001128dbdd5b512aeb9d9a89963aff70b0b5d49e02c5f4: Status 404 returned error can't find the container with id 09e1ea3686db34e3b8001128dbdd5b512aeb9d9a89963aff70b0b5d49e02c5f4 Jul 17 00:17:10.191681 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.191651 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 00:17:10.449571 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.449472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:10.449571 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.449533 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:10.449755 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:10.449634 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:10.449755 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:10.449640 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:10.449755 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:10.449695 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs podName:dc74d918-a560-4e0b-831f-c09d6a1826e9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:11.449677924 +0000 UTC m=+4.196186958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs") pod "network-metrics-daemon-75rcn" (UID: "dc74d918-a560-4e0b-831f-c09d6a1826e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:10.449755 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:10.449709 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret podName:dfabc064-bfdc-4cb0-923c-eb32d6ab392b nodeName:}" failed. No retries permitted until 2026-07-17 00:17:11.449702856 +0000 UTC m=+4.196211887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret") pod "global-pull-secret-syncer-n78sr" (UID: "dfabc064-bfdc-4cb0-923c-eb32d6ab392b") : object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:10.491411 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:10.491378 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c093d7_478e_4edf_820c_34720b639a43.slice/crio-e2a80275499231b77d54021986aa77f255606e6e5c83c114f611c652b7119619 WatchSource:0}: Error finding container e2a80275499231b77d54021986aa77f255606e6e5c83c114f611c652b7119619: Status 404 returned error can't find the container with id e2a80275499231b77d54021986aa77f255606e6e5c83c114f611c652b7119619 Jul 17 00:17:10.550181 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.550147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:10.550360 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:10.550338 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 00:17:10.550427 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:10.550367 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 00:17:10.550427 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:10.550379 2572 projected.go:196] Error preparing data for projected volume kube-api-access-bkkcw for pod openshift-network-diagnostics/network-check-target-xvfn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:10.550518 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:10.550436 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw podName:419c05e1-c598-4835-b870-69336fd5dea9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:11.55041737 +0000 UTC m=+4.296926399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkkcw" (UniqueName: "kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw") pod "network-check-target-xvfn6" (UID: "419c05e1-c598-4835-b870-69336fd5dea9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:10.736443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.736198 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 00:12:09 +0000 UTC" deadline="2027-12-12 09:48:46.635314592 +0000 UTC" Jul 17 00:17:10.736443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:10.736307 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12321h31m35.899014444s" Jul 17 00:17:11.083365 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.082095 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:11.083365 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.082243 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:11.083365 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.082648 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:11.083365 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.082756 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:11.083365 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.082814 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:11.083365 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.082911 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:11.119575 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.119534 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mf9bv" event={"ID":"1faae122-db93-4c37-bf65-16ed121ad6dc","Type":"ContainerStarted","Data":"3fc38393f670c34b7c03d5bbabaa3a99cb5047ad29206e7dad5487e5ede5a3c1"} Jul 17 00:17:11.129270 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.129230 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m676v" event={"ID":"1bd7118e-95e9-46b8-9e06-2579ca5607c3","Type":"ContainerStarted","Data":"09e1ea3686db34e3b8001128dbdd5b512aeb9d9a89963aff70b0b5d49e02c5f4"} Jul 17 00:17:11.146614 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.146577 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" event={"ID":"6b254ab0-677e-4fd2-b0a2-ea3e76928696","Type":"ContainerStarted","Data":"f17066572fff0d298c38598e2d146583ce1169259eddbb9f90ff04c013dad570"} Jul 17 00:17:11.171821 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.171718 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9tj4h" event={"ID":"180b2640-39c5-4cf3-98c6-62deb22ebd32","Type":"ContainerStarted","Data":"4f73720a4121b6edee0d9aea6f3fb0daef4377c0f3263461898100e088cecb77"} Jul 17 00:17:11.198174 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.198131 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cx5p6" event={"ID":"80c093d7-478e-4edf-820c-34720b639a43","Type":"ContainerStarted","Data":"e2a80275499231b77d54021986aa77f255606e6e5c83c114f611c652b7119619"} Jul 17 00:17:11.221080 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.220991 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" event={"ID":"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6","Type":"ContainerStarted","Data":"11fe2f25b45ad367b6f76151153481f0c1f86248b970b262db7ea3df8e2f7494"} Jul 17 00:17:11.460013 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.459162 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:11.460013 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.459261 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:11.460013 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.459391 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:11.460013 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.459456 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs podName:dc74d918-a560-4e0b-831f-c09d6a1826e9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:13.459436573 +0000 UTC m=+6.205945589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs") pod "network-metrics-daemon-75rcn" (UID: "dc74d918-a560-4e0b-831f-c09d6a1826e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:11.460013 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.459840 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:11.460013 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.459910 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret podName:dfabc064-bfdc-4cb0-923c-eb32d6ab392b nodeName:}" failed. No retries permitted until 2026-07-17 00:17:13.459874399 +0000 UTC m=+6.206383411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret") pod "global-pull-secret-syncer-n78sr" (UID: "dfabc064-bfdc-4cb0-923c-eb32d6ab392b") : object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:11.560689 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.560626 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:11.560860 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.560818 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 00:17:11.560860 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.560838 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 00:17:11.560860 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.560852 2572 projected.go:196] Error preparing data for projected volume kube-api-access-bkkcw for pod openshift-network-diagnostics/network-check-target-xvfn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:11.561060 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:11.560940 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw podName:419c05e1-c598-4835-b870-69336fd5dea9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:13.5609097 +0000 UTC m=+6.307418737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkkcw" (UniqueName: "kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw") pod "network-check-target-xvfn6" (UID: "419c05e1-c598-4835-b870-69336fd5dea9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:11.736916 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.736755 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 00:12:09 +0000 UTC" deadline="2028-01-20 12:45:54.953228463 +0000 UTC" Jul 17 00:17:11.736916 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:11.736797 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13260h28m43.216435934s" Jul 17 00:17:13.081915 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:13.081814 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:13.082359 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.081954 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:13.082435 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:13.082374 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:13.082755 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.082474 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:13.082755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:13.082576 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:13.082755 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.082684 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:13.475949 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:13.475854 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:13.476115 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:13.475985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:13.476115 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.476034 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:13.476115 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.476089 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:13.476115 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.476109 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs podName:dc74d918-a560-4e0b-831f-c09d6a1826e9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:17.476089657 +0000 UTC m=+10.222598674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs") pod "network-metrics-daemon-75rcn" (UID: "dc74d918-a560-4e0b-831f-c09d6a1826e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:13.476334 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.476140 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret podName:dfabc064-bfdc-4cb0-923c-eb32d6ab392b nodeName:}" failed. No retries permitted until 2026-07-17 00:17:17.47612431 +0000 UTC m=+10.222633335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret") pod "global-pull-secret-syncer-n78sr" (UID: "dfabc064-bfdc-4cb0-923c-eb32d6ab392b") : object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:13.576907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:13.576342 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:13.576907 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.576511 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 00:17:13.576907 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.576532 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 00:17:13.576907 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.576546 2572 projected.go:196] Error preparing data for projected volume kube-api-access-bkkcw for pod openshift-network-diagnostics/network-check-target-xvfn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:13.576907 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:13.576606 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw podName:419c05e1-c598-4835-b870-69336fd5dea9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:17.576587365 +0000 UTC m=+10.323096387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkkcw" (UniqueName: "kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw") pod "network-check-target-xvfn6" (UID: "419c05e1-c598-4835-b870-69336fd5dea9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:15.082294 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:15.082256 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:15.082294 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:15.082283 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:15.082827 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:15.082250 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:15.082827 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:15.082398 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:15.082827 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:15.082446 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:15.082827 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:15.082516 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:17.081856 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:17.081823 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:17.081856 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:17.081852 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:17.081856 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:17.081823 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:17.082495 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.081990 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:17.082495 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.082206 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:17.082495 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.082272 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:17.508999 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:17.508899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:17.508999 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:17.508989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:17.509229 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.509029 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:17.509229 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.509103 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:17.509229 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.509117 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs podName:dc74d918-a560-4e0b-831f-c09d6a1826e9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:25.509094867 +0000 UTC m=+18.255603886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs") pod "network-metrics-daemon-75rcn" (UID: "dc74d918-a560-4e0b-831f-c09d6a1826e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:17.509229 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.509148 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret podName:dfabc064-bfdc-4cb0-923c-eb32d6ab392b nodeName:}" failed. No retries permitted until 2026-07-17 00:17:25.509133137 +0000 UTC m=+18.255642149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret") pod "global-pull-secret-syncer-n78sr" (UID: "dfabc064-bfdc-4cb0-923c-eb32d6ab392b") : object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:17.610164 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:17.610122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:17.610356 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.610299 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 00:17:17.610356 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.610317 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 00:17:17.610356 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.610329 2572 projected.go:196] Error preparing data for projected volume kube-api-access-bkkcw for pod openshift-network-diagnostics/network-check-target-xvfn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:17.610518 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:17.610392 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw podName:419c05e1-c598-4835-b870-69336fd5dea9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:25.610373952 +0000 UTC m=+18.356882978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkkcw" (UniqueName: "kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw") pod "network-check-target-xvfn6" (UID: "419c05e1-c598-4835-b870-69336fd5dea9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:19.081982 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:19.081946 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:19.082465 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:19.081950 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:19.082465 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:19.082089 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:19.082465 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:19.082182 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:19.082465 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:19.081950 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:19.082465 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:19.082282 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:21.082034 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:21.081993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:21.082452 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:21.081993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:21.082452 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:21.082139 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:21.082452 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:21.081993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:21.082452 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:21.082221 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:21.082452 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:21.082268 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:23.081816 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:23.081775 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:23.082302 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:23.081781 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:23.082302 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:23.081938 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:23.082302 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:23.081936 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:23.082302 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:23.082039 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:23.082302 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:23.082112 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:25.081809 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:25.081771 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:25.082291 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:25.081783 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:25.082291 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.081906 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:25.082291 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:25.081913 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:25.082291 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.082014 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:25.082291 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.082111 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:25.568642 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:25.568542 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:25.568642 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:25.568622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:25.568876 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.568730 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:25.568876 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.568749 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:25.568876 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.568814 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs podName:dc74d918-a560-4e0b-831f-c09d6a1826e9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:41.568790984 +0000 UTC m=+34.315300012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs") pod "network-metrics-daemon-75rcn" (UID: "dc74d918-a560-4e0b-831f-c09d6a1826e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:25.568876 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.568836 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret podName:dfabc064-bfdc-4cb0-923c-eb32d6ab392b nodeName:}" failed. No retries permitted until 2026-07-17 00:17:41.568825294 +0000 UTC m=+34.315334320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret") pod "global-pull-secret-syncer-n78sr" (UID: "dfabc064-bfdc-4cb0-923c-eb32d6ab392b") : object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:25.668977 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:25.668942 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:25.669151 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.669129 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 00:17:25.669194 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.669157 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 00:17:25.669194 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.669168 2572 projected.go:196] Error preparing data for projected volume kube-api-access-bkkcw for pod openshift-network-diagnostics/network-check-target-xvfn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:25.669258 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:25.669223 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw podName:419c05e1-c598-4835-b870-69336fd5dea9 nodeName:}" failed. No retries permitted until 2026-07-17 00:17:41.669208351 +0000 UTC m=+34.415717363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkkcw" (UniqueName: "kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw") pod "network-check-target-xvfn6" (UID: "419c05e1-c598-4835-b870-69336fd5dea9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:27.082207 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:27.082173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:27.082681 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:27.082173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:27.082681 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:27.082300 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:27.082681 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:27.082173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:27.082681 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:27.082408 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:27.082681 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:27.082507 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:29.082073 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:29.082035 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:29.082555 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:29.082109 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:29.082555 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:29.082203 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:29.082555 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:29.082217 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:29.082555 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:29.082314 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:29.082555 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:29.082395 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:31.082501 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.082255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:31.083303 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.082269 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:31.083303 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:31.082615 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:31.083303 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.082278 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:31.083303 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:31.082700 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:31.083303 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:31.082733 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:31.268421 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.268389 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" event={"ID":"9c41058aa23e327c16b34dda6898223f","Type":"ContainerStarted","Data":"78a77c25459b7b1e90f9a5da11ecf0f5e5d859723b1a98c592afd5704f777ac3"} Jul 17 00:17:31.270939 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.270914 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" event={"ID":"16fd2ba4-d022-4232-938c-ee9ab79b44f9","Type":"ContainerStarted","Data":"b9da91b5c6af7c1b0ce3320358b32e3cf0c8205697c0bc4195a9c58a858c681d"} Jul 17 00:17:31.271070 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.270992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" event={"ID":"16fd2ba4-d022-4232-938c-ee9ab79b44f9","Type":"ContainerStarted","Data":"a6a7dfd4ef2607a1a29d224e99db654705732ad661f918b78935517dfc4f068e"} Jul 17 00:17:31.271070 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.271045 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" event={"ID":"16fd2ba4-d022-4232-938c-ee9ab79b44f9","Type":"ContainerStarted","Data":"fc2c216d56bb18e50718c67a232e4cc9d545b2a855fab6de32b2cbb8edadeef7"} Jul 17 00:17:31.271186 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.271090 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" event={"ID":"16fd2ba4-d022-4232-938c-ee9ab79b44f9","Type":"ContainerStarted","Data":"b28827dddfea33c76fd6a38ca625a17704ff811aa97a79cd42ddd243968efd42"} Jul 17 00:17:31.271186 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.271142 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" event={"ID":"16fd2ba4-d022-4232-938c-ee9ab79b44f9","Type":"ContainerStarted","Data":"dd6765b1841cb9a49d443e9a577a55387129a55a6a28feea30d2ea711e176835"} Jul 17 00:17:31.272445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.272422 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cx5p6" event={"ID":"80c093d7-478e-4edf-820c-34720b639a43","Type":"ContainerStarted","Data":"82aa7a608c41415d4cb9e351f0d1ae26f193b9538bb0786b4f34d65a9a7a86a8"} Jul 17 00:17:31.273747 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.273725 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" event={"ID":"1fcc2c34-a1aa-4286-a4d8-c3b1b1fb8af6","Type":"ContainerStarted","Data":"81ba08c939b61128f95d7733b5d74c45be7518a62e96170cb3c665c9845cc783"} Jul 17 00:17:31.275251 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.275231 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mf9bv" event={"ID":"1faae122-db93-4c37-bf65-16ed121ad6dc","Type":"ContainerStarted","Data":"f7ee3c703cb610f0c0c79ef3b01dc784fa81ac22783e1b6c1504cfebf942503c"} Jul 17 00:17:31.276436 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.276416 2572 generic.go:356] "Generic (PLEG): container finished" podID="29420c943b7433d6268f022fb62a1398" containerID="0be0dd3eea869396ef8a36b833f98e04f76e646e285cd471e2f25e85c40f08d9" exitCode=0 Jul 17 00:17:31.276516 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.276470 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" event={"ID":"29420c943b7433d6268f022fb62a1398","Type":"ContainerDied","Data":"0be0dd3eea869396ef8a36b833f98e04f76e646e285cd471e2f25e85c40f08d9"} Jul 17 00:17:31.278587 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.278206 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m676v" event={"ID":"1bd7118e-95e9-46b8-9e06-2579ca5607c3","Type":"ContainerStarted","Data":"576fe045de38a9e6564aca0793c09e144db837037c4aaee3ae06edd09896f277"} Jul 17 00:17:31.280516 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.280493 2572 generic.go:356] "Generic (PLEG): container finished" podID="6b254ab0-677e-4fd2-b0a2-ea3e76928696" containerID="251f7d76caa782fb49573707bb04cc6a47cfed7f0bf97a7abbd4c3fa2bea83c3" exitCode=0 Jul 17 00:17:31.280616 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.280558 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" event={"ID":"6b254ab0-677e-4fd2-b0a2-ea3e76928696","Type":"ContainerDied","Data":"251f7d76caa782fb49573707bb04cc6a47cfed7f0bf97a7abbd4c3fa2bea83c3"} Jul 17 00:17:31.281944 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.281916 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9tj4h" event={"ID":"180b2640-39c5-4cf3-98c6-62deb22ebd32","Type":"ContainerStarted","Data":"81882df6d724093cf040b93a8b84a3741b7862a9d439c29a33348c0cea3c0440"} Jul 17 00:17:31.283357 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.283331 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" event={"ID":"16326435-7285-4778-a104-38a9df454e3c","Type":"ContainerStarted","Data":"c5bd5d5d0f649039d8dd367787c253f319671f21722b256039d63dba9b76c31e"} Jul 17 00:17:31.283444 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.283360 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" podStartSLOduration=22.283341602 podStartE2EDuration="22.283341602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:31.282486729 +0000 UTC m=+24.028995763" watchObservedRunningTime="2026-07-17 00:17:31.283341602 +0000 UTC m=+24.029850637" Jul 17 00:17:31.296977 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.296880 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dvkxv" podStartSLOduration=2.9910363970000002 podStartE2EDuration="23.296864413s" totalImagesPullingTime="20.305828016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:31.296514286 +0000 UTC m=+24.043023298" watchObservedRunningTime="2026-07-17 00:17:31.296864413 +0000 UTC m=+24.043373446" Jul 17 00:17:31.328530 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.328482 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m676v" podStartSLOduration=3.051120357 podStartE2EDuration="23.328465037s" totalImagesPullingTime="20.27734468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:31.327828156 +0000 UTC m=+24.074337191" watchObservedRunningTime="2026-07-17 00:17:31.328465037 +0000 UTC m=+24.074974073" Jul 17 00:17:31.343305 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.343251 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cx5p6" podStartSLOduration=3.229125571 podStartE2EDuration="23.34323711s" totalImagesPullingTime="20.114111539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:31.342870912 +0000 UTC m=+24.089379946" watchObservedRunningTime="2026-07-17 00:17:31.34323711 +0000 UTC m=+24.089746144" Jul 17 00:17:31.356754 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.356693 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9tj4h" podStartSLOduration=3.02137462 podStartE2EDuration="23.356670554s" totalImagesPullingTime="20.335295934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:31.356429725 +0000 UTC m=+24.102938759" watchObservedRunningTime="2026-07-17 00:17:31.356670554 +0000 UTC m=+24.103179588" Jul 17 00:17:31.368873 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:31.368816 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mf9bv" podStartSLOduration=3.050557531 podStartE2EDuration="23.368804215s" totalImagesPullingTime="20.318246684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:31.368612156 +0000 UTC m=+24.115121190" watchObservedRunningTime="2026-07-17 00:17:31.368804215 +0000 UTC m=+24.115313249" Jul 17 00:17:32.279982 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:32.279789 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 00:17:32.286640 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:32.286609 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qk4c4" event={"ID":"2bfe148b-94d1-4c6b-9edf-aa448a273ba1","Type":"ContainerStarted","Data":"94d62ccf1a31582c8e174b1fd30d267ab644df26c570c1be4d5b90692a1f44c0"} Jul 17 00:17:32.288460 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:32.288433 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" event={"ID":"16326435-7285-4778-a104-38a9df454e3c","Type":"ContainerStarted","Data":"14a1e3e8000da164caf9b7a3c42fa96c2312c0866185cc83ffd6cd2de6d48146"} Jul 17 00:17:32.291654 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:32.291627 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" event={"ID":"16fd2ba4-d022-4232-938c-ee9ab79b44f9","Type":"ContainerStarted","Data":"38507f09a6a48cfc33b12ae8a16e23d5c1f9d6fb1bc2c0b6f3d1d58cbd745c23"} Jul 17 00:17:32.299718 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:32.299681 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qk4c4" podStartSLOduration=3.947654037 podStartE2EDuration="24.29966734s" totalImagesPullingTime="20.352013303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:32.299523007 +0000 UTC m=+25.046032041" watchObservedRunningTime="2026-07-17 00:17:32.29966734 +0000 UTC m=+25.046176374" Jul 17 00:17:32.910283 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:32.910186 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T00:17:32.279976004Z","UUID":"1a8d098a-0ed7-404f-95ce-0930dd93f49a","Handler":null,"Name":"","Endpoint":""} Jul 17 00:17:32.911932 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:32.911908 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 00:17:32.912069 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:32.911953 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 00:17:33.081870 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:33.081835 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:33.081870 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:33.081869 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:33.082083 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:33.081875 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:33.082083 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:33.081957 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:33.082151 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:33.082075 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:33.082185 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:33.082142 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:33.296304 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:33.296217 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" event={"ID":"29420c943b7433d6268f022fb62a1398","Type":"ContainerStarted","Data":"b0b6f918ffcc77488b6996b2ebc84d3ceae34a91bb8829925bcf510ea5bd9e4d"} Jul 17 00:17:34.304250 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:34.304218 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" event={"ID":"16fd2ba4-d022-4232-938c-ee9ab79b44f9","Type":"ContainerStarted","Data":"67be38d3a0f64853138b39477cc6b2c55ec3a8cc75cd26ffb8a8936933b1ca1e"} Jul 17 00:17:35.081665 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.081584 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:35.081665 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.081606 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:35.081665 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.081622 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:35.081997 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:35.081700 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:35.081997 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:35.081786 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:35.081997 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:35.081921 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:35.222071 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.222023 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:35.309227 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.309197 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" event={"ID":"16326435-7285-4778-a104-38a9df454e3c","Type":"ContainerStarted","Data":"a85e8bb4a339b12457e3b78da13e911d03ba8cc9b36d47b8de344acbf6475689"} Jul 17 00:17:35.331197 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.331161 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:35.331460 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.331417 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxjxq" podStartSLOduration=3.53127404 podStartE2EDuration="27.331403844s" totalImagesPullingTime="23.800129804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:35.331089457 +0000 UTC m=+28.077598510" watchObservedRunningTime="2026-07-17 00:17:35.331403844 +0000 UTC m=+28.077912878" Jul 17 00:17:35.331626 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.331595 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" podStartSLOduration=26.331589917 podStartE2EDuration="26.331589917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:33.310734757 +0000 UTC m=+26.057243792" watchObservedRunningTime="2026-07-17 00:17:35.331589917 +0000 UTC m=+28.078098951" Jul 17 00:17:35.419499 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.419413 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:35.515591 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.515568 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:35.615972 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:35.615935 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:36.092737 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.092453 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:36.093156 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.093137 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:36.315810 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.315762 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" event={"ID":"16fd2ba4-d022-4232-938c-ee9ab79b44f9","Type":"ContainerStarted","Data":"8017b8b31780f927fa7c6f85d670c9cada2109785abc3eeea54c0997f23c5b84"} Jul 17 00:17:36.316400 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.316373 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:36.316513 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.316492 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m676v" Jul 17 00:17:36.355225 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.355123 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" podStartSLOduration=7.958742607 podStartE2EDuration="28.35510828s" totalImagesPullingTime="20.396365673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:36.353700472 +0000 UTC m=+29.100209525" watchObservedRunningTime="2026-07-17 00:17:36.35510828 +0000 UTC m=+29.101617313" Jul 17 00:17:36.425106 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.425072 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:36.527217 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.527179 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:36.626124 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.626039 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:36.691847 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.691814 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:36.972326 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:36.972204 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:37.051063 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.051034 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:37.082397 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.082302 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:37.082397 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.082339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:37.082617 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.082302 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:37.082617 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:37.082434 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:37.082617 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:37.082567 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:37.082801 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:37.082637 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:37.318735 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.318666 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:37.319283 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.318745 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:37.319283 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.318777 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:37.320081 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.320049 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:37.336833 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.336793 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:37.337417 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.337394 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:17:37.501245 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.501213 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:37.610123 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.610062 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:37.705375 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.705347 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:37.839672 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.839637 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:37.939720 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:37.939627 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:38.222630 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.222314 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n78sr"] Jul 17 00:17:38.222745 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.222692 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:38.222846 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:38.222822 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:38.225520 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.225496 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xvfn6"] Jul 17 00:17:38.225630 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.225604 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:38.225706 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:38.225691 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:38.226124 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.226100 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-75rcn"] Jul 17 00:17:38.226230 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.226207 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:38.226309 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:38.226292 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:38.321193 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.321164 2572 generic.go:356] "Generic (PLEG): container finished" podID="6b254ab0-677e-4fd2-b0a2-ea3e76928696" containerID="86d3d453647aea9b2e2c0dc7c0759199793538d96eae007332473d112675c911" exitCode=0 Jul 17 00:17:38.321663 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.321234 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" event={"ID":"6b254ab0-677e-4fd2-b0a2-ea3e76928696","Type":"ContainerDied","Data":"86d3d453647aea9b2e2c0dc7c0759199793538d96eae007332473d112675c911"} Jul 17 00:17:38.612374 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.612288 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:38.821549 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:38.821515 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:39.497696 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:39.497661 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:40.082278 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:40.081315 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:40.082278 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:40.081465 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:40.082554 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:40.082434 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:40.082614 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:40.082584 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:40.082699 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:40.082679 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:40.082807 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:40.082789 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:40.327791 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:40.327701 2572 generic.go:356] "Generic (PLEG): container finished" podID="6b254ab0-677e-4fd2-b0a2-ea3e76928696" containerID="532f9109cc9efb2764cc49323a7bc7d78eeaf59953b825b988e24af1189d4b9d" exitCode=0 Jul 17 00:17:40.327791 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:40.327746 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" event={"ID":"6b254ab0-677e-4fd2-b0a2-ea3e76928696","Type":"ContainerDied","Data":"532f9109cc9efb2764cc49323a7bc7d78eeaf59953b825b988e24af1189d4b9d"} Jul 17 00:17:40.681499 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:40.681414 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:40.781818 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:40.781789 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:41.332936 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:41.332821 2572 generic.go:356] "Generic (PLEG): container finished" podID="6b254ab0-677e-4fd2-b0a2-ea3e76928696" containerID="83753fdb1b29c79baded5437b17e2b7daa71f1626ce5afde9e087bb2f026f581" exitCode=0 Jul 17 00:17:41.332936 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:41.332878 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" event={"ID":"6b254ab0-677e-4fd2-b0a2-ea3e76928696","Type":"ContainerDied","Data":"83753fdb1b29c79baded5437b17e2b7daa71f1626ce5afde9e087bb2f026f581"} Jul 17 00:17:41.586084 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:41.585980 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:41.586084 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:41.586039 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:41.586273 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:41.586124 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:41.586273 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:41.586138 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:41.586273 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:41.586185 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret podName:dfabc064-bfdc-4cb0-923c-eb32d6ab392b nodeName:}" failed. No retries permitted until 2026-07-17 00:18:13.586169453 +0000 UTC m=+66.332678470 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret") pod "global-pull-secret-syncer-n78sr" (UID: "dfabc064-bfdc-4cb0-923c-eb32d6ab392b") : object "kube-system"/"original-pull-secret" not registered Jul 17 00:17:41.586273 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:41.586200 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs podName:dc74d918-a560-4e0b-831f-c09d6a1826e9 nodeName:}" failed. No retries permitted until 2026-07-17 00:18:13.586193411 +0000 UTC m=+66.332702423 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs") pod "network-metrics-daemon-75rcn" (UID: "dc74d918-a560-4e0b-831f-c09d6a1826e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 00:17:41.686975 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:41.686941 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:41.687327 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:41.687097 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 00:17:41.687327 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:41.687117 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 00:17:41.687327 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:41.687126 2572 projected.go:196] Error preparing data for projected volume kube-api-access-bkkcw for pod openshift-network-diagnostics/network-check-target-xvfn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:41.687327 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:41.687176 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw podName:419c05e1-c598-4835-b870-69336fd5dea9 nodeName:}" failed. No retries permitted until 2026-07-17 00:18:13.687163147 +0000 UTC m=+66.433672163 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkkcw" (UniqueName: "kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw") pod "network-check-target-xvfn6" (UID: "419c05e1-c598-4835-b870-69336fd5dea9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 00:17:42.081720 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:42.081689 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:42.081916 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:42.081690 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:42.081916 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:42.081813 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:42.081916 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:42.081825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:42.082069 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:42.081944 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:42.082069 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:42.082044 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:44.081954 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.081926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:44.082294 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.081926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:44.082294 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:44.082044 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75rcn" podUID="dc74d918-a560-4e0b-831f-c09d6a1826e9" Jul 17 00:17:44.082294 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:44.082101 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xvfn6" podUID="419c05e1-c598-4835-b870-69336fd5dea9" Jul 17 00:17:44.082294 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.081926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:44.082294 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:44.082169 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n78sr" podUID="dfabc064-bfdc-4cb0-923c-eb32d6ab392b" Jul 17 00:17:44.516408 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.516376 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeReady" Jul 17 00:17:44.516572 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.516542 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 00:17:44.558783 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.558741 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rlvlj"] Jul 17 00:17:44.585142 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.585101 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7wrvh"] Jul 17 00:17:44.585322 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.585122 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.589262 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.589233 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q9rpn\"" Jul 17 00:17:44.589443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.589356 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 00:17:44.589443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.589377 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 00:17:44.599120 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.599096 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rlvlj"] Jul 17 00:17:44.599254 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.599139 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7wrvh"] Jul 17 00:17:44.599254 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.599162 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2kwdw"] Jul 17 00:17:44.599254 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.599211 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7wrvh" Jul 17 00:17:44.601580 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.601562 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 00:17:44.601698 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.601565 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 00:17:44.601698 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.601565 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 00:17:44.601698 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.601640 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mxbsx\"" Jul 17 00:17:44.608083 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.608048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6xc\" (UniqueName: \"kubernetes.io/projected/6382649c-69a4-455a-9879-9e275333eb0f-kube-api-access-4c6xc\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.608208 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.608105 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52mnn\" (UniqueName: \"kubernetes.io/projected/30a743cd-32ba-43eb-8805-991f080b23b4-kube-api-access-52mnn\") pod \"ingress-canary-7wrvh\" (UID: \"30a743cd-32ba-43eb-8805-991f080b23b4\") " pod="openshift-ingress-canary/ingress-canary-7wrvh" Jul 17 00:17:44.608960 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.608821 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6382649c-69a4-455a-9879-9e275333eb0f-tmp-dir\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.608960 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.608866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6382649c-69a4-455a-9879-9e275333eb0f-config-volume\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.608960 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.608882 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6382649c-69a4-455a-9879-9e275333eb0f-metrics-tls\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.608960 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.608924 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30a743cd-32ba-43eb-8805-991f080b23b4-cert\") pod \"ingress-canary-7wrvh\" (UID: \"30a743cd-32ba-43eb-8805-991f080b23b4\") " pod="openshift-ingress-canary/ingress-canary-7wrvh" Jul 17 00:17:44.609945 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.609916 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:44.616589 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.616552 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2kwdw"] Jul 17 00:17:44.616735 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.616680 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.619356 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.619264 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 00:17:44.619356 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.619308 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 00:17:44.619871 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.619626 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mrnf5\"" Jul 17 00:17:44.619871 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.619653 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 00:17:44.619871 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.619714 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 00:17:44.703301 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.703270 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:44.709480 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709446 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6xc\" (UniqueName: \"kubernetes.io/projected/6382649c-69a4-455a-9879-9e275333eb0f-kube-api-access-4c6xc\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.709649 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52mnn\" (UniqueName: \"kubernetes.io/projected/30a743cd-32ba-43eb-8805-991f080b23b4-kube-api-access-52mnn\") pod \"ingress-canary-7wrvh\" (UID: \"30a743cd-32ba-43eb-8805-991f080b23b4\") " pod="openshift-ingress-canary/ingress-canary-7wrvh" Jul 17 00:17:44.709649 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709536 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/203fcf71-4593-418d-9bd1-f2e2876a071b-crio-socket\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.709649 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709593 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgzz\" (UniqueName: \"kubernetes.io/projected/203fcf71-4593-418d-9bd1-f2e2876a071b-kube-api-access-shgzz\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.709821 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6382649c-69a4-455a-9879-9e275333eb0f-tmp-dir\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.709821 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709793 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/203fcf71-4593-418d-9bd1-f2e2876a071b-data-volume\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.709960 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709819 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/203fcf71-4593-418d-9bd1-f2e2876a071b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.709960 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6382649c-69a4-455a-9879-9e275333eb0f-config-volume\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.709960 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6382649c-69a4-455a-9879-9e275333eb0f-metrics-tls\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.710101 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709943 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30a743cd-32ba-43eb-8805-991f080b23b4-cert\") pod \"ingress-canary-7wrvh\" (UID: \"30a743cd-32ba-43eb-8805-991f080b23b4\") " pod="openshift-ingress-canary/ingress-canary-7wrvh" Jul 17 00:17:44.710101 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.709993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/203fcf71-4593-418d-9bd1-f2e2876a071b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.710206 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.710132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6382649c-69a4-455a-9879-9e275333eb0f-tmp-dir\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.710456 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.710432 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6382649c-69a4-455a-9879-9e275333eb0f-config-volume\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.714740 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.714591 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6382649c-69a4-455a-9879-9e275333eb0f-metrics-tls\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.714854 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.714779 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30a743cd-32ba-43eb-8805-991f080b23b4-cert\") pod \"ingress-canary-7wrvh\" (UID: \"30a743cd-32ba-43eb-8805-991f080b23b4\") " pod="openshift-ingress-canary/ingress-canary-7wrvh" Jul 17 00:17:44.720268 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.720243 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52mnn\" (UniqueName: \"kubernetes.io/projected/30a743cd-32ba-43eb-8805-991f080b23b4-kube-api-access-52mnn\") pod \"ingress-canary-7wrvh\" (UID: \"30a743cd-32ba-43eb-8805-991f080b23b4\") " pod="openshift-ingress-canary/ingress-canary-7wrvh" Jul 17 00:17:44.720400 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.720365 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6xc\" (UniqueName: \"kubernetes.io/projected/6382649c-69a4-455a-9879-9e275333eb0f-kube-api-access-4c6xc\") pod \"dns-default-rlvlj\" (UID: \"6382649c-69a4-455a-9879-9e275333eb0f\") " pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.804125 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.804001 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:44.811052 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.811024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/203fcf71-4593-418d-9bd1-f2e2876a071b-data-volume\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.811215 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.811058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/203fcf71-4593-418d-9bd1-f2e2876a071b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.811215 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.811093 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/203fcf71-4593-418d-9bd1-f2e2876a071b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.811337 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.811241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/203fcf71-4593-418d-9bd1-f2e2876a071b-crio-socket\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.811337 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.811288 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgzz\" (UniqueName: \"kubernetes.io/projected/203fcf71-4593-418d-9bd1-f2e2876a071b-kube-api-access-shgzz\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.811458 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.811433 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/203fcf71-4593-418d-9bd1-f2e2876a071b-data-volume\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.811520 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.811486 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/203fcf71-4593-418d-9bd1-f2e2876a071b-crio-socket\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.811742 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.811719 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/203fcf71-4593-418d-9bd1-f2e2876a071b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.813665 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.813643 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/203fcf71-4593-418d-9bd1-f2e2876a071b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.820521 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.820493 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgzz\" (UniqueName: \"kubernetes.io/projected/203fcf71-4593-418d-9bd1-f2e2876a071b-kube-api-access-shgzz\") pod \"insights-runtime-extractor-2kwdw\" (UID: \"203fcf71-4593-418d-9bd1-f2e2876a071b\") " pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:44.896672 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.896612 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:44.909851 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.909810 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7wrvh" Jul 17 00:17:44.928050 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:44.928017 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2kwdw" Jul 17 00:17:46.081946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.081904 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:17:46.082446 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.082124 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:17:46.082851 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.081904 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:17:46.086242 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.086202 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 00:17:46.087443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.086799 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24tr\"" Jul 17 00:17:46.087443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.087037 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 00:17:46.087443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.087214 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 00:17:46.087443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.087386 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvh75\"" Jul 17 00:17:46.087443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.087437 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 00:17:46.223902 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.223853 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rlvlj"] Jul 17 00:17:46.226381 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.226350 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2kwdw"] Jul 17 00:17:46.226864 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.226844 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7wrvh"] Jul 17 00:17:46.280985 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:46.280939 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6382649c_69a4_455a_9879_9e275333eb0f.slice/crio-65ff961d3005b4f19dde5afe929713a23d76b69e2a7e818cb10b287086cb9052 WatchSource:0}: Error finding container 65ff961d3005b4f19dde5afe929713a23d76b69e2a7e818cb10b287086cb9052: Status 404 returned error can't find the container with id 65ff961d3005b4f19dde5afe929713a23d76b69e2a7e818cb10b287086cb9052 Jul 17 00:17:46.281581 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:46.281473 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a743cd_32ba_43eb_8805_991f080b23b4.slice/crio-dfd622cc94cbae83d30d2d9ba9391d0a02a4f2b391c3f19f378010e5a78a3c52 WatchSource:0}: Error finding container dfd622cc94cbae83d30d2d9ba9391d0a02a4f2b391c3f19f378010e5a78a3c52: Status 404 returned error can't find the container with id dfd622cc94cbae83d30d2d9ba9391d0a02a4f2b391c3f19f378010e5a78a3c52 Jul 17 00:17:46.282342 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:46.282314 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203fcf71_4593_418d_9bd1_f2e2876a071b.slice/crio-baa92a624f11aa27c02c50119729b4cb6e34ed942148d312f050543538a3e9f4 WatchSource:0}: Error finding container baa92a624f11aa27c02c50119729b4cb6e34ed942148d312f050543538a3e9f4: Status 404 returned error can't find the container with id baa92a624f11aa27c02c50119729b4cb6e34ed942148d312f050543538a3e9f4 Jul 17 00:17:46.350007 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.349913 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2kwdw" event={"ID":"203fcf71-4593-418d-9bd1-f2e2876a071b","Type":"ContainerStarted","Data":"baa92a624f11aa27c02c50119729b4cb6e34ed942148d312f050543538a3e9f4"} Jul 17 00:17:46.352386 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.352351 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlvlj" event={"ID":"6382649c-69a4-455a-9879-9e275333eb0f","Type":"ContainerStarted","Data":"65ff961d3005b4f19dde5afe929713a23d76b69e2a7e818cb10b287086cb9052"} Jul 17 00:17:46.353673 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:46.353647 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7wrvh" event={"ID":"30a743cd-32ba-43eb-8805-991f080b23b4","Type":"ContainerStarted","Data":"dfd622cc94cbae83d30d2d9ba9391d0a02a4f2b391c3f19f378010e5a78a3c52"} Jul 17 00:17:47.360928 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:47.360645 2572 generic.go:356] "Generic (PLEG): container finished" podID="6b254ab0-677e-4fd2-b0a2-ea3e76928696" containerID="d6f7539e2518038e62a551e92ce1e01a2d3775e43f44b38d83bbab8c509859a6" exitCode=0 Jul 17 00:17:47.361777 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:47.360779 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" event={"ID":"6b254ab0-677e-4fd2-b0a2-ea3e76928696","Type":"ContainerDied","Data":"d6f7539e2518038e62a551e92ce1e01a2d3775e43f44b38d83bbab8c509859a6"} Jul 17 00:17:47.731623 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:47.731478 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:47.834105 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:47.834070 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:48.113242 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.113186 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:48.222298 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.222232 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:48.228064 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.228039 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x"] Jul 17 00:17:48.263321 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.263281 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x"] Jul 17 00:17:48.263485 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.263399 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" Jul 17 00:17:48.266136 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.265978 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 17 00:17:48.266136 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.266013 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fhdz4\"" Jul 17 00:17:48.287225 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.287155 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:48.341533 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.341492 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a0e59b55-7716-422c-a323-297e95c569bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-glp6x\" (UID: \"a0e59b55-7716-422c-a323-297e95c569bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" Jul 17 00:17:48.349540 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.349502 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:48.368708 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.368628 2572 generic.go:356] "Generic (PLEG): container finished" podID="6b254ab0-677e-4fd2-b0a2-ea3e76928696" containerID="0c169aadc17c921a6bfcc4a03d179810e61f7ad4aad77f48c18ed6094dad61d7" exitCode=0 Jul 17 00:17:48.369174 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.368713 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" event={"ID":"6b254ab0-677e-4fd2-b0a2-ea3e76928696","Type":"ContainerDied","Data":"0c169aadc17c921a6bfcc4a03d179810e61f7ad4aad77f48c18ed6094dad61d7"} Jul 17 00:17:48.412373 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.412294 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:48.443426 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.442751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a0e59b55-7716-422c-a323-297e95c569bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-glp6x\" (UID: \"a0e59b55-7716-422c-a323-297e95c569bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" Jul 17 00:17:48.443664 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:48.443454 2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 17 00:17:48.443664 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:17:48.443527 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e59b55-7716-422c-a323-297e95c569bc-tls-certificates podName:a0e59b55-7716-422c-a323-297e95c569bc nodeName:}" failed. No retries permitted until 2026-07-17 00:17:48.943505813 +0000 UTC m=+41.690014829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a0e59b55-7716-422c-a323-297e95c569bc-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-glp6x" (UID: "a0e59b55-7716-422c-a323-297e95c569bc") : secret "prometheus-operator-admission-webhook-tls" not found Jul 17 00:17:48.546927 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.546851 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:48.947331 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.947269 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a0e59b55-7716-422c-a323-297e95c569bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-glp6x\" (UID: \"a0e59b55-7716-422c-a323-297e95c569bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" Jul 17 00:17:48.951122 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:48.951096 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a0e59b55-7716-422c-a323-297e95c569bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-glp6x\" (UID: \"a0e59b55-7716-422c-a323-297e95c569bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" Jul 17 00:17:49.034718 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:49.034690 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:49.085116 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:49.085079 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:49.176509 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:49.176483 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" Jul 17 00:17:49.375650 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:49.375622 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" event={"ID":"6b254ab0-677e-4fd2-b0a2-ea3e76928696","Type":"ContainerStarted","Data":"e1d12a168e376b3c0d76d1f3df9a4cf34ad7e57d570eb21a5e84b8c44f110a29"} Jul 17 00:17:49.397864 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:49.397629 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8fbm4" podStartSLOduration=7.604077167 podStartE2EDuration="41.397591005s" totalImagesPullingTime="33.793513838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:49.395801895 +0000 UTC m=+42.142310965" watchObservedRunningTime="2026-07-17 00:17:49.397591005 +0000 UTC m=+42.144100040" Jul 17 00:17:49.521994 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:49.521907 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:49.617666 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:49.617632 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:51.026834 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.026803 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-796b6495d-gpsbp"] Jul 17 00:17:51.240907 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.240860 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796b6495d-gpsbp"] Jul 17 00:17:51.241090 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.241001 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.243659 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.243636 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 00:17:51.243784 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.243664 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 00:17:51.244811 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.244795 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 00:17:51.245031 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.245011 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9gjx9\"" Jul 17 00:17:51.245031 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.245017 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 00:17:51.245207 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.245056 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 00:17:51.245207 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.245136 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 00:17:51.245330 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.245267 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 00:17:51.246379 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.246331 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:51.250308 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.250290 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 00:17:51.265599 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.265574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-oauth-config\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.265693 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.265629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-oauth-serving-cert\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.265693 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.265662 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-service-ca\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.265786 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.265693 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-serving-cert\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.265786 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.265710 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjllx\" (UniqueName: \"kubernetes.io/projected/92b088cb-58ec-48be-8fbb-7e28edbbda10-kube-api-access-rjllx\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.265786 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.265745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-config\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.265786 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.265769 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-trusted-ca-bundle\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.304162 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.304110 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:51.366359 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.366331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-serving-cert\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.366500 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.366367 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjllx\" (UniqueName: \"kubernetes.io/projected/92b088cb-58ec-48be-8fbb-7e28edbbda10-kube-api-access-rjllx\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.366500 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.366417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-config\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.366500 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.366442 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-trusted-ca-bundle\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.366500 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.366476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-oauth-config\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.371258 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.366564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-oauth-serving-cert\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.371258 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.367052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-service-ca\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.371258 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.367745 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-config\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.371258 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.368599 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-oauth-serving-cert\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.371258 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.368582 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-service-ca\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.371258 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.368810 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-trusted-ca-bundle\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.372285 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.372258 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-oauth-config\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.373178 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.373156 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-serving-cert\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.375065 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.375048 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjllx\" (UniqueName: \"kubernetes.io/projected/92b088cb-58ec-48be-8fbb-7e28edbbda10-kube-api-access-rjllx\") pod \"console-796b6495d-gpsbp\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:51.551204 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:51.551169 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:17:53.431935 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:53.431912 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x"] Jul 17 00:17:53.451952 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:53.451918 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796b6495d-gpsbp"] Jul 17 00:17:53.490172 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:53.490114 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e59b55_7716_422c_a323_297e95c569bc.slice/crio-9c80616e6dbbec071d088ebf92a7238f42da5b837c0bcac2f003f5f3d10db87f WatchSource:0}: Error finding container 9c80616e6dbbec071d088ebf92a7238f42da5b837c0bcac2f003f5f3d10db87f: Status 404 returned error can't find the container with id 9c80616e6dbbec071d088ebf92a7238f42da5b837c0bcac2f003f5f3d10db87f Jul 17 00:17:53.490659 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:17:53.490637 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b088cb_58ec_48be_8fbb_7e28edbbda10.slice/crio-65e53c380dfe8f12ed7ad387c19d2a34e9e33c529daee40d24c727886458e1af WatchSource:0}: Error finding container 65e53c380dfe8f12ed7ad387c19d2a34e9e33c529daee40d24c727886458e1af: Status 404 returned error can't find the container with id 65e53c380dfe8f12ed7ad387c19d2a34e9e33c529daee40d24c727886458e1af Jul 17 00:17:54.393659 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:54.393565 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7wrvh" event={"ID":"30a743cd-32ba-43eb-8805-991f080b23b4","Type":"ContainerStarted","Data":"ccd4204c08e060bf967c72bd24547cd1c09bbfc24c592735f4b6151b001fb397"} Jul 17 00:17:54.397236 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:54.397198 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796b6495d-gpsbp" event={"ID":"92b088cb-58ec-48be-8fbb-7e28edbbda10","Type":"ContainerStarted","Data":"65e53c380dfe8f12ed7ad387c19d2a34e9e33c529daee40d24c727886458e1af"} Jul 17 00:17:54.399268 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:54.399239 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" event={"ID":"a0e59b55-7716-422c-a323-297e95c569bc","Type":"ContainerStarted","Data":"9c80616e6dbbec071d088ebf92a7238f42da5b837c0bcac2f003f5f3d10db87f"} Jul 17 00:17:54.402736 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:54.402653 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2kwdw" event={"ID":"203fcf71-4593-418d-9bd1-f2e2876a071b","Type":"ContainerStarted","Data":"0c54005eb3067679ed70df49652eb8bde06f72495aa004ec721b891936dd906e"} Jul 17 00:17:54.407401 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:54.406795 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlvlj" event={"ID":"6382649c-69a4-455a-9879-9e275333eb0f","Type":"ContainerStarted","Data":"50edadb00c5424fb4d078c494c05a9bc0fd2bbd2bcd44cba1934626f11794661"} Jul 17 00:17:54.407401 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:54.406845 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlvlj" event={"ID":"6382649c-69a4-455a-9879-9e275333eb0f","Type":"ContainerStarted","Data":"e5dbbbf82b0893c5068df4c39949c918dadc7027944bbe0ea25d73063142b4bc"} Jul 17 00:17:54.407401 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:54.407268 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rlvlj" Jul 17 00:17:54.411819 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:54.411303 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7wrvh" podStartSLOduration=3.41052725 podStartE2EDuration="10.411287023s" totalImagesPullingTime="7.000759773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:54.40985954 +0000 UTC m=+47.156368652" watchObservedRunningTime="2026-07-17 00:17:54.411287023 +0000 UTC m=+47.157796035" Jul 17 00:17:54.430493 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:54.430435 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rlvlj" podStartSLOduration=3.439360615 podStartE2EDuration="10.430413703s" totalImagesPullingTime="6.991053088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:54.430220058 +0000 UTC m=+47.176729093" watchObservedRunningTime="2026-07-17 00:17:54.430413703 +0000 UTC m=+47.176922736" Jul 17 00:17:55.412792 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:55.412753 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" event={"ID":"a0e59b55-7716-422c-a323-297e95c569bc","Type":"ContainerStarted","Data":"fa05e6147a69eb606f00397c2a354ff799c6f0b8b961663d69b95796b500666a"} Jul 17 00:17:55.413625 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:55.413603 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" Jul 17 00:17:55.419449 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:55.419394 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" Jul 17 00:17:55.449235 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:55.449172 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-glp6x" podStartSLOduration=5.735390104 podStartE2EDuration="7.449147145s" totalImagesPullingTime="1.713757041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:55.447556235 +0000 UTC m=+48.194065270" watchObservedRunningTime="2026-07-17 00:17:55.449147145 +0000 UTC m=+48.195656180" Jul 17 00:17:56.252539 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:56.252499 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:56.317194 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:56.317156 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:56.591909 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:56.591813 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:58.305373 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:58.305333 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:58.406080 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:58.406048 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:58.502323 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:58.502291 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:58.558448 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:58.558366 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:59.427112 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:59.427083 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796b6495d-gpsbp" event={"ID":"92b088cb-58ec-48be-8fbb-7e28edbbda10","Type":"ContainerStarted","Data":"83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d"} Jul 17 00:17:59.444528 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:59.444478 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-796b6495d-gpsbp" podStartSLOduration=2.76886389 podStartE2EDuration="8.44446237s" totalImagesPullingTime="5.67559848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:17:59.444045609 +0000 UTC m=+52.190554643" watchObservedRunningTime="2026-07-17 00:17:59.44446237 +0000 UTC m=+52.190971402" Jul 17 00:17:59.592104 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:59.592069 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:17:59.652474 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:17:59.652442 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:00.432603 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:00.432558 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2kwdw" event={"ID":"203fcf71-4593-418d-9bd1-f2e2876a071b","Type":"ContainerStarted","Data":"ea4c0d23c9becd109e384714b0fb1ade6794b4f4f4328e619f7f8c4c5452dac1"} Jul 17 00:18:00.638659 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:00.638624 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:00.739856 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:00.739773 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:01.454061 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:01.454018 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:01.507451 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:01.507419 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:01.551389 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:01.551360 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:18:01.551566 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:01.551426 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:18:01.556802 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:01.556775 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:18:02.444904 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:02.444864 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:18:02.588317 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:02.588280 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:02.644785 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:02.644743 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:02.699065 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:02.698989 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:03.310196 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.310158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:03.411351 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.411268 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:03.449284 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.449251 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2kwdw" event={"ID":"203fcf71-4593-418d-9bd1-f2e2876a071b","Type":"ContainerStarted","Data":"db7f766ab0d158c6808c8a656ee1f443963667bae04ddcfe5c0f0d325c5c4bc6"} Jul 17 00:18:03.467157 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.467104 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2kwdw" podStartSLOduration=2.82551359 podStartE2EDuration="19.467088716s" totalImagesPullingTime="16.641575126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:03.465511294 +0000 UTC m=+56.212020349" watchObservedRunningTime="2026-07-17 00:18:03.467088716 +0000 UTC m=+56.213597792" Jul 17 00:18:03.602796 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.602763 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:03.609483 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.609455 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww"] Jul 17 00:18:03.612059 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.612043 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.614317 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.614278 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 00:18:03.614507 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.614483 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 00:18:03.614615 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.614583 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 00:18:03.614756 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.614733 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 00:18:03.614964 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.614945 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 00:18:03.615088 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.615032 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-68mbc\"" Jul 17 00:18:03.622343 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.622323 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww"] Jul 17 00:18:03.636314 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.636283 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vhr26"] Jul 17 00:18:03.638654 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.638616 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fm782"] Jul 17 00:18:03.638794 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.638724 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.640684 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.640665 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.641147 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.641126 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 00:18:03.641147 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.641134 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 00:18:03.641314 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.641305 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4sxfj\"" Jul 17 00:18:03.641381 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.641366 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 00:18:03.642857 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.642837 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 00:18:03.642955 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.642840 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 00:18:03.643137 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.643106 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2g2fw\"" Jul 17 00:18:03.643233 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.643165 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 00:18:03.649100 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.649079 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vhr26"] Jul 17 00:18:03.665791 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.665719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/77419013-35d6-4961-8303-d0b49a55de56-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.665940 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.665791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77419013-35d6-4961-8303-d0b49a55de56-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.665940 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.665818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwzx\" (UniqueName: \"kubernetes.io/projected/77419013-35d6-4961-8303-d0b49a55de56-kube-api-access-kbwzx\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.665940 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.665842 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77419013-35d6-4961-8303-d0b49a55de56-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.694832 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.694796 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:03.767088 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767049 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5gw\" (UniqueName: \"kubernetes.io/projected/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-kube-api-access-6d5gw\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.767088 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767087 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-tls\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.767296 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767105 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.767296 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767160 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77419013-35d6-4961-8303-d0b49a55de56-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.767296 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767194 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.767296 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767234 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-textfile\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.767296 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767272 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8lmg\" (UniqueName: \"kubernetes.io/projected/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-api-access-r8lmg\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.767444 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767316 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/77419013-35d6-4961-8303-d0b49a55de56-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.767444 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.767444 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:18:03.767397 2572 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 17 00:18:03.767444 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-sys\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.767444 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767435 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-root\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.767660 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:18:03.767450 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77419013-35d6-4961-8303-d0b49a55de56-openshift-state-metrics-tls podName:77419013-35d6-4961-8303-d0b49a55de56 nodeName:}" failed. No retries permitted until 2026-07-17 00:18:04.267436075 +0000 UTC m=+57.013945088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/77419013-35d6-4961-8303-d0b49a55de56-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-hvrww" (UID: "77419013-35d6-4961-8303-d0b49a55de56") : secret "openshift-state-metrics-tls" not found Jul 17 00:18:03.767660 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767482 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f896ad14-cb87-46cd-9eb9-13b10971743e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.767660 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767515 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-wtmp\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.767660 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767536 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f896ad14-cb87-46cd-9eb9-13b10971743e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.767660 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767582 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwzx\" (UniqueName: \"kubernetes.io/projected/77419013-35d6-4961-8303-d0b49a55de56-kube-api-access-kbwzx\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.767660 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767612 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-metrics-client-ca\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.767660 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77419013-35d6-4961-8303-d0b49a55de56-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.768004 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767677 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.768004 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767728 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.768004 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.767859 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77419013-35d6-4961-8303-d0b49a55de56-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.769935 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.769917 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77419013-35d6-4961-8303-d0b49a55de56-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.775976 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.775957 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwzx\" (UniqueName: \"kubernetes.io/projected/77419013-35d6-4961-8303-d0b49a55de56-kube-api-access-kbwzx\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:03.794064 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.794034 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:03.868236 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868202 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868426 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868248 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-textfile\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868490 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lmg\" (UniqueName: \"kubernetes.io/projected/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-api-access-r8lmg\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.868546 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868513 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.868546 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868533 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-textfile\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868543 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-sys\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868571 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-root\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868594 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f896ad14-cb87-46cd-9eb9-13b10971743e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.868656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868618 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-wtmp\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868622 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-sys\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868635 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-root\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868978 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868643 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f896ad14-cb87-46cd-9eb9-13b10971743e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.868978 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868705 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-metrics-client-ca\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868978 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868978 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868770 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-wtmp\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868978 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.868978 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868824 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5gw\" (UniqueName: \"kubernetes.io/projected/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-kube-api-access-6d5gw\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868978 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868850 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-tls\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.868978 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.868876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.869368 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.869001 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f896ad14-cb87-46cd-9eb9-13b10971743e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.869368 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:18:03.869009 2572 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 17 00:18:03.869368 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:18:03.869096 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-tls podName:f896ad14-cb87-46cd-9eb9-13b10971743e nodeName:}" failed. No retries permitted until 2026-07-17 00:18:04.36907761 +0000 UTC m=+57.115586636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-vhr26" (UID: "f896ad14-cb87-46cd-9eb9-13b10971743e") : secret "kube-state-metrics-tls" not found Jul 17 00:18:03.869368 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.869254 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.869368 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.869306 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-metrics-client-ca\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.869368 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.869361 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.869676 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.869445 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f896ad14-cb87-46cd-9eb9-13b10971743e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.870839 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.870815 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.871402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.871375 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-node-exporter-tls\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.871402 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.871396 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.876790 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.876768 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5gw\" (UniqueName: \"kubernetes.io/projected/0b5ecc4a-f010-4fbe-95b7-792af5cef2f7-kube-api-access-6d5gw\") pod \"node-exporter-fm782\" (UID: \"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7\") " pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.876877 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.876836 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8lmg\" (UniqueName: \"kubernetes.io/projected/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-api-access-r8lmg\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:03.894713 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.894682 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:03.954713 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.954628 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fm782" Jul 17 00:18:03.964548 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:03.964522 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5ecc4a_f010_4fbe_95b7_792af5cef2f7.slice/crio-8cd11798ef1ca0ce13e7d38e437851f2bf5e269ba34967726ac728241ebd3178 WatchSource:0}: Error finding container 8cd11798ef1ca0ce13e7d38e437851f2bf5e269ba34967726ac728241ebd3178: Status 404 returned error can't find the container with id 8cd11798ef1ca0ce13e7d38e437851f2bf5e269ba34967726ac728241ebd3178 Jul 17 00:18:03.997631 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:03.997600 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:04.096404 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.096352 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:04.197079 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.197026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:04.272927 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.272802 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/77419013-35d6-4961-8303-d0b49a55de56-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:04.275272 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.275250 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/77419013-35d6-4961-8303-d0b49a55de56-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hvrww\" (UID: \"77419013-35d6-4961-8303-d0b49a55de56\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:04.297339 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.297309 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:04.374113 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.374073 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:04.376510 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.376486 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f896ad14-cb87-46cd-9eb9-13b10971743e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vhr26\" (UID: \"f896ad14-cb87-46cd-9eb9-13b10971743e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:04.397929 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.397882 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:04.415555 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.415528 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rlvlj" Jul 17 00:18:04.454080 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.454046 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fm782" event={"ID":"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7","Type":"ContainerStarted","Data":"8cd11798ef1ca0ce13e7d38e437851f2bf5e269ba34967726ac728241ebd3178"} Jul 17 00:18:04.520558 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.520518 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" Jul 17 00:18:04.549123 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.549094 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" Jul 17 00:18:04.601833 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.601734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:04.674857 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.674825 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww"] Jul 17 00:18:04.678376 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.678332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:04.678709 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:04.678680 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77419013_35d6_4961_8303_d0b49a55de56.slice/crio-5f59cb9a5bc5f14e490f0f7f8c80f5dd84bc640866dd14452ed172b21beeef79 WatchSource:0}: Error finding container 5f59cb9a5bc5f14e490f0f7f8c80f5dd84bc640866dd14452ed172b21beeef79: Status 404 returned error can't find the container with id 5f59cb9a5bc5f14e490f0f7f8c80f5dd84bc640866dd14452ed172b21beeef79 Jul 17 00:18:04.692586 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.692558 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 00:18:04.696099 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.696077 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.701447 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.700601 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 00:18:04.701447 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.700672 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 00:18:04.701447 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.700837 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 00:18:04.701447 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.701124 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 00:18:04.702507 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.702371 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 00:18:04.702663 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.702648 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 00:18:04.703054 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.703035 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 00:18:04.705127 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.705007 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q6kqs\"" Jul 17 00:18:04.705127 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.705053 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vhr26"] Jul 17 00:18:04.705127 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.705127 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 00:18:04.709481 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.709257 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 00:18:04.710947 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:04.710919 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf896ad14_cb87_46cd_9eb9_13b10971743e.slice/crio-7db5f8ec7d51bd9eafd2601732678b6ae3b7f4f1433ea5a9a7076208db24b993 WatchSource:0}: Error finding container 7db5f8ec7d51bd9eafd2601732678b6ae3b7f4f1433ea5a9a7076208db24b993: Status 404 returned error can't find the container with id 7db5f8ec7d51bd9eafd2601732678b6ae3b7f4f1433ea5a9a7076208db24b993 Jul 17 00:18:04.712004 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.711985 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 00:18:04.778171 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778141 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778361 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778338 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-web-config\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778478 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778388 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778478 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7nb\" (UniqueName: \"kubernetes.io/projected/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-kube-api-access-md7nb\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778591 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778496 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778591 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778535 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778591 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778730 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778602 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778730 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778639 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778730 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778665 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778730 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778708 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778939 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778749 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-config-out\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.778939 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.778801 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.803235 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.803132 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:04.879252 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879219 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879262 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-web-config\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879292 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879445 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7nb\" (UniqueName: \"kubernetes.io/projected/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-kube-api-access-md7nb\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879475 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879506 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879558 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879835 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879618 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879835 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879660 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879835 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879701 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-config-out\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.879835 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.879749 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.881012 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.880137 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.881012 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.880198 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.881012 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.880951 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.882871 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.882716 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.883087 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.883065 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-web-config\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.883160 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.883091 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.883209 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.883191 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.883409 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.883390 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.883562 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.883544 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.884390 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.884368 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.884726 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.884708 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.884854 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.884839 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-config-out\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:04.887969 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:04.887874 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7nb\" (UniqueName: \"kubernetes.io/projected/22bb06a1-bac7-4bfc-b5b0-1efc004de6c5-kube-api-access-md7nb\") pod \"alertmanager-main-0\" (UID: \"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:05.035470 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.035386 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 00:18:05.129726 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.129635 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:05.181523 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.181482 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 00:18:05.223073 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:05.223034 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bb06a1_bac7_4bfc_b5b0_1efc004de6c5.slice/crio-aa4e73a02a879905a8830bb5fd23c9ca7bca796fd641daf44e2c87c9b12fe071 WatchSource:0}: Error finding container aa4e73a02a879905a8830bb5fd23c9ca7bca796fd641daf44e2c87c9b12fe071: Status 404 returned error can't find the container with id aa4e73a02a879905a8830bb5fd23c9ca7bca796fd641daf44e2c87c9b12fe071 Jul 17 00:18:05.461357 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.461314 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" event={"ID":"77419013-35d6-4961-8303-d0b49a55de56","Type":"ContainerStarted","Data":"1c2aa03ea8a89931685b597ccc11c91f4c54c53cef5d5d4fc14f186b9998adfd"} Jul 17 00:18:05.461661 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.461391 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" event={"ID":"77419013-35d6-4961-8303-d0b49a55de56","Type":"ContainerStarted","Data":"415a82075886e0e20979783e6d44ab17547d4ba80a57e5c122b93b147a26fbb3"} Jul 17 00:18:05.461661 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.461424 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" event={"ID":"77419013-35d6-4961-8303-d0b49a55de56","Type":"ContainerStarted","Data":"5f59cb9a5bc5f14e490f0f7f8c80f5dd84bc640866dd14452ed172b21beeef79"} Jul 17 00:18:05.462836 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.462774 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" event={"ID":"f896ad14-cb87-46cd-9eb9-13b10971743e","Type":"ContainerStarted","Data":"7db5f8ec7d51bd9eafd2601732678b6ae3b7f4f1433ea5a9a7076208db24b993"} Jul 17 00:18:05.464893 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.464865 2572 generic.go:356] "Generic (PLEG): container finished" podID="0b5ecc4a-f010-4fbe-95b7-792af5cef2f7" containerID="14078cfbf1ff9ad4bdddb876d473453e6aaa08cfe0d30ab5bf076f4ee2dd8a6f" exitCode=0 Jul 17 00:18:05.465006 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.464942 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fm782" event={"ID":"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7","Type":"ContainerDied","Data":"14078cfbf1ff9ad4bdddb876d473453e6aaa08cfe0d30ab5bf076f4ee2dd8a6f"} Jul 17 00:18:05.466654 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.466567 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5","Type":"ContainerStarted","Data":"aa4e73a02a879905a8830bb5fd23c9ca7bca796fd641daf44e2c87c9b12fe071"} Jul 17 00:18:05.816498 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:05.816417 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:06.471303 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.471269 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fm782" event={"ID":"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7","Type":"ContainerStarted","Data":"5ba7b67c37246d261d4cc3c24e134fa1ab0d6c67f7d9ce807fdafea7e05bf843"} Jul 17 00:18:06.471435 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.471322 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fm782" event={"ID":"0b5ecc4a-f010-4fbe-95b7-792af5cef2f7","Type":"ContainerStarted","Data":"6523fbc82d285682eb8763ec4b8db2884e259252beeaeab093d78e0299109911"} Jul 17 00:18:06.495753 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.495708 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fm782" podStartSLOduration=2.350860885 podStartE2EDuration="3.495693804s" totalImagesPullingTime="1.075727006s" totalInitContainerRuntime="69.105913ms" isStatefulPod=true podCreationTimestamp="2026-07-17 00:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:06.49445281 +0000 UTC m=+59.240961847" watchObservedRunningTime="2026-07-17 00:18:06.495693804 +0000 UTC m=+59.242202816" Jul 17 00:18:06.565081 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.565053 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:06.620211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.620178 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9d89d8974-zdlqm"] Jul 17 00:18:06.624503 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.624438 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.627605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.627211 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 00:18:06.627605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.627242 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 00:18:06.627605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.627313 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 00:18:06.627605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.627371 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4cbr2\"" Jul 17 00:18:06.627605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.627414 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9c4ci61c423b8\"" Jul 17 00:18:06.627605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.627495 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 00:18:06.627605 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.627537 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 00:18:06.637972 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.637862 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9d89d8974-zdlqm"] Jul 17 00:18:06.668945 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.668811 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:06.696153 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.696123 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.696311 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.696170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-grpc-tls\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.696311 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.696238 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxft\" (UniqueName: \"kubernetes.io/projected/0a404540-3048-48c6-a355-b93bed008a7e-kube-api-access-qsxft\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.696311 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.696287 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.696446 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.696354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.696446 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.696373 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-tls\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.696446 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.696419 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a404540-3048-48c6-a355-b93bed008a7e-metrics-client-ca\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.696588 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.696479 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.797149 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.797110 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.797358 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.797163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-tls\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.797358 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.797226 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a404540-3048-48c6-a355-b93bed008a7e-metrics-client-ca\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.797358 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.797267 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.797358 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.797305 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.797358 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.797341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-grpc-tls\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.797609 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.797368 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxft\" (UniqueName: \"kubernetes.io/projected/0a404540-3048-48c6-a355-b93bed008a7e-kube-api-access-qsxft\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.797609 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.797406 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.798111 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.798049 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a404540-3048-48c6-a355-b93bed008a7e-metrics-client-ca\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.800293 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.800266 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-tls\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.800489 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.800293 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.800489 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.800428 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-grpc-tls\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.800489 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.800433 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.800777 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.800752 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.801150 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.801129 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a404540-3048-48c6-a355-b93bed008a7e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.806023 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.806001 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxft\" (UniqueName: \"kubernetes.io/projected/0a404540-3048-48c6-a355-b93bed008a7e-kube-api-access-qsxft\") pod \"thanos-querier-9d89d8974-zdlqm\" (UID: \"0a404540-3048-48c6-a355-b93bed008a7e\") " pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:06.919442 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.919354 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:06.937552 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:06.937520 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:07.076396 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.076363 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9d89d8974-zdlqm"] Jul 17 00:18:07.080204 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:07.080180 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a404540_3048_48c6_a355_b93bed008a7e.slice/crio-4c9c697b0d09d37a58bae727449d8ce607ad179bc8b05567c629d781485a4e04 WatchSource:0}: Error finding container 4c9c697b0d09d37a58bae727449d8ce607ad179bc8b05567c629d781485a4e04: Status 404 returned error can't find the container with id 4c9c697b0d09d37a58bae727449d8ce607ad179bc8b05567c629d781485a4e04 Jul 17 00:18:07.143045 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.143013 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:07.476604 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.476570 2572 generic.go:356] "Generic (PLEG): container finished" podID="22bb06a1-bac7-4bfc-b5b0-1efc004de6c5" containerID="e09fe48fd885164a3c5384d5d3b0da7befb52b35cb12248218df0f6e764080ef" exitCode=0 Jul 17 00:18:07.476803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.476669 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5","Type":"ContainerDied","Data":"e09fe48fd885164a3c5384d5d3b0da7befb52b35cb12248218df0f6e764080ef"} Jul 17 00:18:07.477935 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.477912 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" event={"ID":"0a404540-3048-48c6-a355-b93bed008a7e","Type":"ContainerStarted","Data":"4c9c697b0d09d37a58bae727449d8ce607ad179bc8b05567c629d781485a4e04"} Jul 17 00:18:07.480248 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.480226 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" event={"ID":"77419013-35d6-4961-8303-d0b49a55de56","Type":"ContainerStarted","Data":"09786426b0bffe5e0cd387f7cde62c40c4de80fcef8bf8cb1fc0a492ab746be8"} Jul 17 00:18:07.482394 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.482368 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" event={"ID":"f896ad14-cb87-46cd-9eb9-13b10971743e","Type":"ContainerStarted","Data":"adf8d55f4ed3e33f607f3306d1f4165a44c1be58ad29fe91650cf5263b7907dc"} Jul 17 00:18:07.482473 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.482424 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" event={"ID":"f896ad14-cb87-46cd-9eb9-13b10971743e","Type":"ContainerStarted","Data":"9c8bf0a7a090ee76c17db98cf9e784c0cd8098206a3063e0c0912bb6f6c35095"} Jul 17 00:18:07.482473 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.482455 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" event={"ID":"f896ad14-cb87-46cd-9eb9-13b10971743e","Type":"ContainerStarted","Data":"6996798ce5c41042c7b8b0b04d06bf1e46f1a101c877f8efeacbd61a8a52805f"} Jul 17 00:18:07.522623 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.522567 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hvrww" podStartSLOduration=2.900138579 podStartE2EDuration="4.522549346s" totalImagesPullingTime="1.622410767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:07.520333352 +0000 UTC m=+60.266842385" watchObservedRunningTime="2026-07-17 00:18:07.522549346 +0000 UTC m=+60.269058379" Jul 17 00:18:07.553881 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:07.553826 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-vhr26" podStartSLOduration=2.834191588 podStartE2EDuration="4.553808422s" totalImagesPullingTime="1.719616834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:07.5519529 +0000 UTC m=+60.298461946" watchObservedRunningTime="2026-07-17 00:18:07.553808422 +0000 UTC m=+60.300317434" Jul 17 00:18:08.462829 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.462784 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:08.464760 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.464727 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8555f8f758-9mm99"] Jul 17 00:18:08.485755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.485727 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.492074 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.492033 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8555f8f758-9mm99"] Jul 17 00:18:08.547693 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.547658 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:08.614252 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.614203 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lfms\" (UniqueName: \"kubernetes.io/projected/2722bd21-96ee-446f-ba16-3f23403c297f-kube-api-access-9lfms\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.614252 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.614241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-console-config\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.614510 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.614269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-trusted-ca-bundle\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.614510 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.614405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-serving-cert\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.614741 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.614670 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-oauth-config\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.614868 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.614852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-oauth-serving-cert\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.615066 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.615041 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-service-ca\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.647618 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.647544 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:08.716307 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.716207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lfms\" (UniqueName: \"kubernetes.io/projected/2722bd21-96ee-446f-ba16-3f23403c297f-kube-api-access-9lfms\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.716307 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.716254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-console-config\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.716307 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.716280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-trusted-ca-bundle\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.716595 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.716460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-serving-cert\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.716595 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.716547 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-oauth-config\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.716595 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.716575 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-oauth-serving-cert\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.716726 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.716620 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-service-ca\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.717134 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.717103 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-console-config\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.717259 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.717232 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-trusted-ca-bundle\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.717317 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.717271 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-service-ca\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.717374 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.717337 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-oauth-serving-cert\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.719258 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.719234 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-oauth-config\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.719420 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.719388 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-serving-cert\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.724020 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.723995 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lfms\" (UniqueName: \"kubernetes.io/projected/2722bd21-96ee-446f-ba16-3f23403c297f-kube-api-access-9lfms\") pod \"console-8555f8f758-9mm99\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.804757 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.804714 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:08.839571 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.839538 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-669c5bb98c-t9nbj"] Jul 17 00:18:08.855243 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.855207 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:08.855741 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.855708 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-669c5bb98c-t9nbj"] Jul 17 00:18:08.858195 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.858172 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 00:18:08.858490 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.858461 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 00:18:08.858670 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.858650 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 00:18:08.858836 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.858821 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cx568\"" Jul 17 00:18:08.859052 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.859037 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 00:18:08.859333 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.859257 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 00:18:08.863430 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.863405 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 00:18:08.919421 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.919369 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-telemeter-client-tls\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:08.919421 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.919421 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-metrics-client-ca\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:08.919656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.919486 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-secret-telemeter-client\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:08.919656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.919523 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:08.919656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.919594 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:08.919656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.919627 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-serving-certs-ca-bundle\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:08.919656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.919651 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-federate-client-tls\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:08.919863 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.919740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpmsj\" (UniqueName: \"kubernetes.io/projected/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-kube-api-access-zpmsj\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:08.986591 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:08.986506 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:09.020655 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.020618 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-telemeter-client-tls\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.020655 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.020657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-metrics-client-ca\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.020962 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.020751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-secret-telemeter-client\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.020962 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.020786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.020962 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.020806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.020962 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.020835 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-serving-certs-ca-bundle\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.020962 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.020861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-federate-client-tls\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.021211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.020981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpmsj\" (UniqueName: \"kubernetes.io/projected/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-kube-api-access-zpmsj\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.021504 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.021454 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-metrics-client-ca\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.022066 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.021995 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-serving-certs-ca-bundle\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.022066 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.022040 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.023912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.023842 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-telemeter-client-tls\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.024022 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.023982 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-secret-telemeter-client\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.024022 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.024003 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-federate-client-tls\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.024406 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.024383 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.031426 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.031402 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpmsj\" (UniqueName: \"kubernetes.io/projected/281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c-kube-api-access-zpmsj\") pod \"telemeter-client-669c5bb98c-t9nbj\" (UID: \"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c\") " pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.168289 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.168252 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" Jul 17 00:18:09.335107 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.335033 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qbd7" Jul 17 00:18:09.440216 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.440176 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:09.542794 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.541535 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:09.647318 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.646722 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:09.648938 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.648914 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8555f8f758-9mm99"] Jul 17 00:18:09.663828 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:09.663419 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2722bd21_96ee_446f_ba16_3f23403c297f.slice/crio-907e611ff4760607a5831aa391279591059c6f35860c24082a926609dd1e6444 WatchSource:0}: Error finding container 907e611ff4760607a5831aa391279591059c6f35860c24082a926609dd1e6444: Status 404 returned error can't find the container with id 907e611ff4760607a5831aa391279591059c6f35860c24082a926609dd1e6444 Jul 17 00:18:09.686860 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.686833 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-669c5bb98c-t9nbj"] Jul 17 00:18:09.859331 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.859234 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:09.863599 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.863571 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 00:18:09.867453 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.867428 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.870186 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.870159 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 00:18:09.870312 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.870288 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 00:18:09.871141 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.870822 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 00:18:09.871229 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.871205 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 00:18:09.871448 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.871427 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 00:18:09.871527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.871463 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 00:18:09.871711 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.871693 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 00:18:09.871794 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.871777 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-776g4\"" Jul 17 00:18:09.871944 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.871699 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 00:18:09.871994 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.871941 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 00:18:09.872180 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.872164 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 00:18:09.872731 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.872714 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 00:18:09.873729 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.872753 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 00:18:09.873729 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.872771 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 00:18:09.873729 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.872852 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 00:18:09.873729 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.873072 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-84k4ci6f1u4hr\"" Jul 17 00:18:09.874545 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.874529 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 00:18:09.886350 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.886256 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 00:18:09.931881 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.931856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-config\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932035 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.931905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932035 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.931997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932035 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932024 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mprj\" (UniqueName: \"kubernetes.io/projected/0068c298-8e29-472f-8e33-318812b54dbc-kube-api-access-8mprj\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932208 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932054 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932208 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932115 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932208 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932151 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932208 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932178 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932208 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932202 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932434 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932228 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932434 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932434 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932365 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932434 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932389 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0068c298-8e29-472f-8e33-318812b54dbc-config-out\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932434 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932673 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932440 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932673 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932468 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932673 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932499 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-web-config\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932673 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932524 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932673 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932547 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0068c298-8e29-472f-8e33-318812b54dbc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:09.932673 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:09.932573 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.033701 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.033662 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.033915 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.033722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.033915 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.033742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0068c298-8e29-472f-8e33-318812b54dbc-config-out\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.033915 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.033759 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.033915 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.033776 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.033915 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.033795 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.033915 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.033817 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-web-config\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.033915 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.033839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.033915 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.033864 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0068c298-8e29-472f-8e33-318812b54dbc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034298 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034121 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034298 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-config\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034298 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034248 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034469 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034469 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034366 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mprj\" (UniqueName: \"kubernetes.io/projected/0068c298-8e29-472f-8e33-318812b54dbc-kube-api-access-8mprj\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034469 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034469 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034658 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034658 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034499 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034658 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034526 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034658 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034857 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034729 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.034928 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.034882 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.035582 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.035529 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.035723 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.035674 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.036505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.036250 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.036505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.036365 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.037092 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.037068 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.037174 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.037108 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.037688 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.037663 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-web-config\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.038047 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.038022 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0068c298-8e29-472f-8e33-318812b54dbc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.038749 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.038694 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0068c298-8e29-472f-8e33-318812b54dbc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.039094 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.039064 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.039348 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.039321 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.039436 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.039321 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.039562 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.039541 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0068c298-8e29-472f-8e33-318812b54dbc-config-out\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.039670 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.039645 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.039811 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.039794 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.040122 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.040097 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-config\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.040262 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.040243 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0068c298-8e29-472f-8e33-318812b54dbc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.042014 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.041994 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mprj\" (UniqueName: \"kubernetes.io/projected/0068c298-8e29-472f-8e33-318812b54dbc-kube-api-access-8mprj\") pod \"prometheus-k8s-0\" (UID: \"0068c298-8e29-472f-8e33-318812b54dbc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.211938 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.211863 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:10.357706 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.357674 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 00:18:10.361755 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:10.361654 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0068c298_8e29_472f_8e33_318812b54dbc.slice/crio-c991ef2aec9c70643ffce91639e16e93c50acd94e6632c7a776192b559ffe5a8 WatchSource:0}: Error finding container c991ef2aec9c70643ffce91639e16e93c50acd94e6632c7a776192b559ffe5a8: Status 404 returned error can't find the container with id c991ef2aec9c70643ffce91639e16e93c50acd94e6632c7a776192b559ffe5a8 Jul 17 00:18:10.508034 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.507944 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5","Type":"ContainerStarted","Data":"5ff7676636ae2132c8aa295efd672726dca7f5b65d1ee69d60c7fb28e82315f6"} Jul 17 00:18:10.508359 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.508325 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5","Type":"ContainerStarted","Data":"1afef331d5e9f81c6bf81257b614d13066e62adff28ff8bc9ccd8d04281de47f"} Jul 17 00:18:10.508487 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.508385 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5","Type":"ContainerStarted","Data":"62bc677d4330568ff09f32c65b3f61ab04a62900d3f021d1e4a36910dc091436"} Jul 17 00:18:10.508487 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.508425 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5","Type":"ContainerStarted","Data":"a1754cfe9792f922b0fa5a5503e290f18b65a4d458edcd4809dce2e7e580eef3"} Jul 17 00:18:10.508487 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.508464 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5","Type":"ContainerStarted","Data":"bfc97231a07ef186097f6c6430efd6d8f9e0dc4664081b7e7b8d7188145a0202"} Jul 17 00:18:10.510495 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.510462 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" event={"ID":"0a404540-3048-48c6-a355-b93bed008a7e","Type":"ContainerStarted","Data":"51cf198451d569a8ac24837103956342b39647140d03db03c23291979750027d"} Jul 17 00:18:10.510617 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.510543 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" event={"ID":"0a404540-3048-48c6-a355-b93bed008a7e","Type":"ContainerStarted","Data":"adee5f4afa67229220ce49e2a18daf378e947131dd5c5b2542b3fff937cae663"} Jul 17 00:18:10.510617 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.510586 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" event={"ID":"0a404540-3048-48c6-a355-b93bed008a7e","Type":"ContainerStarted","Data":"51d1daac809bb9ac62c4f408d6ccea935c49ef915cb5b465bed7f1c7243ad413"} Jul 17 00:18:10.512231 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.512204 2572 generic.go:356] "Generic (PLEG): container finished" podID="0068c298-8e29-472f-8e33-318812b54dbc" containerID="7da9b34a20d72e06c74b882b09a5ca65ad36c24c6ff93c74282529e020cfae5d" exitCode=0 Jul 17 00:18:10.512336 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.512300 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0068c298-8e29-472f-8e33-318812b54dbc","Type":"ContainerDied","Data":"7da9b34a20d72e06c74b882b09a5ca65ad36c24c6ff93c74282529e020cfae5d"} Jul 17 00:18:10.512404 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.512375 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0068c298-8e29-472f-8e33-318812b54dbc","Type":"ContainerStarted","Data":"c991ef2aec9c70643ffce91639e16e93c50acd94e6632c7a776192b559ffe5a8"} Jul 17 00:18:10.513904 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.513859 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" event={"ID":"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c","Type":"ContainerStarted","Data":"e79d1a854127dfcc2197bd10ccc0fbe33af64f3db08e03adbe891998c74cdace"} Jul 17 00:18:10.516219 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.516197 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8555f8f758-9mm99" event={"ID":"2722bd21-96ee-446f-ba16-3f23403c297f","Type":"ContainerStarted","Data":"96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369"} Jul 17 00:18:10.516452 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.516360 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8555f8f758-9mm99" event={"ID":"2722bd21-96ee-446f-ba16-3f23403c297f","Type":"ContainerStarted","Data":"907e611ff4760607a5831aa391279591059c6f35860c24082a926609dd1e6444"} Jul 17 00:18:10.560710 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:10.560631 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8555f8f758-9mm99" podStartSLOduration=2.56060587 podStartE2EDuration="2.56060587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:10.559957359 +0000 UTC m=+63.306466394" watchObservedRunningTime="2026-07-17 00:18:10.56060587 +0000 UTC m=+63.307114905" Jul 17 00:18:11.116001 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:11.115958 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:11.217342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:11.217311 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:11.526246 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:11.526144 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22bb06a1-bac7-4bfc-b5b0-1efc004de6c5","Type":"ContainerStarted","Data":"3e05c2c0673e92ab8216aedf14880cec6d16020d8894ad5f4ad2c4048affd433"} Jul 17 00:18:11.531412 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:11.531378 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" event={"ID":"0a404540-3048-48c6-a355-b93bed008a7e","Type":"ContainerStarted","Data":"3ac5572b6e0b2a050815183afe7bfa4a9282b58e18ca59752b9757d0d96cfebd"} Jul 17 00:18:11.531580 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:11.531464 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" event={"ID":"0a404540-3048-48c6-a355-b93bed008a7e","Type":"ContainerStarted","Data":"a167c3d4d23f139e3eed2bc2d5d276274435d17f0f235bbebf8575811e17469e"} Jul 17 00:18:11.531580 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:11.531501 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" event={"ID":"0a404540-3048-48c6-a355-b93bed008a7e","Type":"ContainerStarted","Data":"5ace25eea89e1308a0a306dfb19e8530743fb2e95d3727ebfc8842a78df8c920"} Jul 17 00:18:11.550842 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:11.550732 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.526668507 podStartE2EDuration="7.550712003s" totalImagesPullingTime="3.993241932s" totalInitContainerRuntime="30.801564ms" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:11.54845291 +0000 UTC m=+64.294961944" watchObservedRunningTime="2026-07-17 00:18:11.550712003 +0000 UTC m=+64.297221037" Jul 17 00:18:11.568235 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:11.568012 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" podStartSLOduration=2.229786775 podStartE2EDuration="5.567988421s" totalImagesPullingTime="3.338201646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:11.567952156 +0000 UTC m=+64.314461216" watchObservedRunningTime="2026-07-17 00:18:11.567988421 +0000 UTC m=+64.314497459" Jul 17 00:18:12.538738 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:12.538693 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" event={"ID":"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c","Type":"ContainerStarted","Data":"cf1bc1b4d27cfdea060be9c632e3d751e2143ab0c2c1cb85c4371dc137152ac1"} Jul 17 00:18:12.538941 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:12.538767 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" event={"ID":"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c","Type":"ContainerStarted","Data":"1217e8793f390e44f0fc9186d6ca2630fa2ae4456638b330b09c1513accd9bd8"} Jul 17 00:18:12.538941 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:12.538800 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" event={"ID":"281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c","Type":"ContainerStarted","Data":"c07f6c0402c6cd61800b8aeb14603e55f35573af469ce4cb3f842044f5b6b690"} Jul 17 00:18:12.539520 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:12.539485 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:12.560297 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:12.560243 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-669c5bb98c-t9nbj" podStartSLOduration=2.599686536 podStartE2EDuration="4.560220715s" totalImagesPullingTime="1.960534179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:12.558969044 +0000 UTC m=+65.305478098" watchObservedRunningTime="2026-07-17 00:18:12.560220715 +0000 UTC m=+65.306729751" Jul 17 00:18:12.857691 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:12.857657 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:12.911588 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:12.911546 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:13.446816 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.446785 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8555f8f758-9mm99"] Jul 17 00:18:13.476172 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.476147 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-866b45cb98-28qc2"] Jul 17 00:18:13.479667 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.479648 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.498763 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.498438 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866b45cb98-28qc2"] Jul 17 00:18:13.547969 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.547785 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0068c298-8e29-472f-8e33-318812b54dbc","Type":"ContainerStarted","Data":"29ad35a58f80492a980bffa9665eacfc60b370693b57616244806aa848997f39"} Jul 17 00:18:13.547969 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.547875 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0068c298-8e29-472f-8e33-318812b54dbc","Type":"ContainerStarted","Data":"6c648589a0c11214e8d50cbb46df4c5131aafce65b096c6b330a20dbdde4e31e"} Jul 17 00:18:13.574803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.574772 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-serving-cert\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.574965 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.574943 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-config\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.575050 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.574982 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-trusted-ca-bundle\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.575111 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.575090 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-oauth-serving-cert\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.575169 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.575156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-oauth-config\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.575218 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.575182 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-service-ca\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.575741 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.575284 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjbp\" (UniqueName: \"kubernetes.io/projected/c5234caf-2e88-4d8e-bd8f-fee075a254d8-kube-api-access-pnjbp\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.676025 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.675997 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-config\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.676136 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.676052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-trusted-ca-bundle\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.676136 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.676122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-oauth-serving-cert\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.676242 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.676177 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-oauth-config\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.676242 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.676203 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-service-ca\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.676444 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.676424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjbp\" (UniqueName: \"kubernetes.io/projected/c5234caf-2e88-4d8e-bd8f-fee075a254d8-kube-api-access-pnjbp\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.677450 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.677423 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-oauth-serving-cert\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.677541 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.677488 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-service-ca\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.677685 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.677661 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-config\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.677931 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.677905 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-serving-cert\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.678015 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.677974 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:18:13.678110 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.678087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:18:13.683951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.680087 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-trusted-ca-bundle\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.683951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.680387 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 00:18:13.683951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.681457 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-oauth-config\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.683951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.682971 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 00:18:13.695064 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.694267 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc74d918-a560-4e0b-831f-c09d6a1826e9-metrics-certs\") pod \"network-metrics-daemon-75rcn\" (UID: \"dc74d918-a560-4e0b-831f-c09d6a1826e9\") " pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:18:13.696264 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.696242 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-serving-cert\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.696492 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.696468 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfabc064-bfdc-4cb0-923c-eb32d6ab392b-original-pull-secret\") pod \"global-pull-secret-syncer-n78sr\" (UID: \"dfabc064-bfdc-4cb0-923c-eb32d6ab392b\") " pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:18:13.698510 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.698484 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjbp\" (UniqueName: \"kubernetes.io/projected/c5234caf-2e88-4d8e-bd8f-fee075a254d8-kube-api-access-pnjbp\") pod \"console-866b45cb98-28qc2\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.699296 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.699276 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvh75\"" Jul 17 00:18:13.707573 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.707552 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75rcn" Jul 17 00:18:13.709344 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.709327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n78sr" Jul 17 00:18:13.779551 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.779473 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:18:13.782329 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.782265 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 00:18:13.792809 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.792532 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 00:18:13.803881 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.803817 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkkcw\" (UniqueName: \"kubernetes.io/projected/419c05e1-c598-4835-b870-69336fd5dea9-kube-api-access-bkkcw\") pod \"network-check-target-xvfn6\" (UID: \"419c05e1-c598-4835-b870-69336fd5dea9\") " pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:18:13.814872 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.814845 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:13.877250 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.877133 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n78sr"] Jul 17 00:18:13.880857 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:13.880794 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfabc064_bfdc_4cb0_923c_eb32d6ab392b.slice/crio-e935ef9443ecb8cb50a3bf0fe4f1e5af102ca3b3d16051764834784f0a7643b8 WatchSource:0}: Error finding container e935ef9443ecb8cb50a3bf0fe4f1e5af102ca3b3d16051764834784f0a7643b8: Status 404 returned error can't find the container with id e935ef9443ecb8cb50a3bf0fe4f1e5af102ca3b3d16051764834784f0a7643b8 Jul 17 00:18:13.898318 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.898250 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-75rcn"] Jul 17 00:18:13.901184 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:13.901145 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc74d918_a560_4e0b_831f_c09d6a1826e9.slice/crio-a46d0e340a3f706787463c60eb2eaf06ccb575c6bd7f468b88256b1650a56395 WatchSource:0}: Error finding container a46d0e340a3f706787463c60eb2eaf06ccb575c6bd7f468b88256b1650a56395: Status 404 returned error can't find the container with id a46d0e340a3f706787463c60eb2eaf06ccb575c6bd7f468b88256b1650a56395 Jul 17 00:18:13.964434 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:13.964408 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866b45cb98-28qc2"] Jul 17 00:18:13.967039 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:13.967011 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5234caf_2e88_4d8e_bd8f_fee075a254d8.slice/crio-a5126ab3296ed49febfaf66465f8047f278e4ccad0db1ced76f162eb4f045446 WatchSource:0}: Error finding container a5126ab3296ed49febfaf66465f8047f278e4ccad0db1ced76f162eb4f045446: Status 404 returned error can't find the container with id a5126ab3296ed49febfaf66465f8047f278e4ccad0db1ced76f162eb4f045446 Jul 17 00:18:14.007098 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.007076 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24tr\"" Jul 17 00:18:14.015331 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.015308 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:18:14.153588 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.153508 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xvfn6"] Jul 17 00:18:14.156519 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:18:14.156493 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419c05e1_c598_4835_b870_69336fd5dea9.slice/crio-7fe8fc3cf41a144e53490b643e8d9e43c9bc44c179030a7038750758c04b997b WatchSource:0}: Error finding container 7fe8fc3cf41a144e53490b643e8d9e43c9bc44c179030a7038750758c04b997b: Status 404 returned error can't find the container with id 7fe8fc3cf41a144e53490b643e8d9e43c9bc44c179030a7038750758c04b997b Jul 17 00:18:14.553758 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.553663 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xvfn6" event={"ID":"419c05e1-c598-4835-b870-69336fd5dea9","Type":"ContainerStarted","Data":"7fe8fc3cf41a144e53490b643e8d9e43c9bc44c179030a7038750758c04b997b"} Jul 17 00:18:14.557128 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.557098 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0068c298-8e29-472f-8e33-318812b54dbc","Type":"ContainerStarted","Data":"94cfa7f5edad3b9a5a81e9d6845dcb16997bf9976d95cd67976ec20a0c806c75"} Jul 17 00:18:14.557256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.557173 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0068c298-8e29-472f-8e33-318812b54dbc","Type":"ContainerStarted","Data":"bb27da839b84cae0dba3f351d64bdcc85ccbac9fa0871d469040bf3c59c25418"} Jul 17 00:18:14.557256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.557217 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0068c298-8e29-472f-8e33-318812b54dbc","Type":"ContainerStarted","Data":"526a81d72222dae3ae79c590c958d25d8bc164290d819d044414bfa93d43f151"} Jul 17 00:18:14.557347 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.557256 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0068c298-8e29-472f-8e33-318812b54dbc","Type":"ContainerStarted","Data":"1ce2057759c13aad665ba92d488593fa808e1770ed83b5ab94e44ae28c2bdd57"} Jul 17 00:18:14.558352 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.558328 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n78sr" event={"ID":"dfabc064-bfdc-4cb0-923c-eb32d6ab392b","Type":"ContainerStarted","Data":"e935ef9443ecb8cb50a3bf0fe4f1e5af102ca3b3d16051764834784f0a7643b8"} Jul 17 00:18:14.559855 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.559836 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866b45cb98-28qc2" event={"ID":"c5234caf-2e88-4d8e-bd8f-fee075a254d8","Type":"ContainerStarted","Data":"3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751"} Jul 17 00:18:14.559940 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.559870 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866b45cb98-28qc2" event={"ID":"c5234caf-2e88-4d8e-bd8f-fee075a254d8","Type":"ContainerStarted","Data":"a5126ab3296ed49febfaf66465f8047f278e4ccad0db1ced76f162eb4f045446"} Jul 17 00:18:14.561033 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.561014 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-75rcn" event={"ID":"dc74d918-a560-4e0b-831f-c09d6a1826e9","Type":"ContainerStarted","Data":"a46d0e340a3f706787463c60eb2eaf06ccb575c6bd7f468b88256b1650a56395"} Jul 17 00:18:14.603341 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.603281 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.709971108 podStartE2EDuration="5.603259864s" totalImagesPullingTime="2.87099195s" totalInitContainerRuntime="22.296806ms" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:14.5996358 +0000 UTC m=+67.346144862" watchObservedRunningTime="2026-07-17 00:18:14.603259864 +0000 UTC m=+67.349768896" Jul 17 00:18:14.626831 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.626769 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-866b45cb98-28qc2" podStartSLOduration=1.626726035 podStartE2EDuration="1.626726035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:14.625594858 +0000 UTC m=+67.372103892" watchObservedRunningTime="2026-07-17 00:18:14.626726035 +0000 UTC m=+67.373235070" Jul 17 00:18:14.629710 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.629681 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:14.719228 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:14.719190 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:15.212184 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:15.212149 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:18:16.580938 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:16.580532 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-75rcn" event={"ID":"dc74d918-a560-4e0b-831f-c09d6a1826e9","Type":"ContainerStarted","Data":"a9d33e88aa75dd5e134ebddac8edc468ac394e0abe059a772b2316fc470e9bb1"} Jul 17 00:18:16.580938 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:16.580628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-75rcn" event={"ID":"dc74d918-a560-4e0b-831f-c09d6a1826e9","Type":"ContainerStarted","Data":"6bbf5104c585eedd5966c661d6a8ffb8ce7963c30f63816c7f4df42b596a212a"} Jul 17 00:18:16.610314 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:16.610211 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-75rcn" podStartSLOduration=66.828772667 podStartE2EDuration="1m8.610193172s" totalImagesPullingTime="1.781420505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:16.609807908 +0000 UTC m=+69.356316943" watchObservedRunningTime="2026-07-17 00:18:16.610193172 +0000 UTC m=+69.356702209" Jul 17 00:18:16.740874 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:16.740830 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:16.840923 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:16.840782 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:18.555432 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:18.555408 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9d89d8974-zdlqm" Jul 17 00:18:18.804836 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:18.804804 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:19.596108 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:19.596072 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xvfn6" event={"ID":"419c05e1-c598-4835-b870-69336fd5dea9","Type":"ContainerStarted","Data":"ae2ffd1fdbe632114e76a2e55f5ac301820b85c663e1db8ad4809091307ca231"} Jul 17 00:18:19.596509 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:19.596176 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:18:19.597599 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:19.597575 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n78sr" event={"ID":"dfabc064-bfdc-4cb0-923c-eb32d6ab392b","Type":"ContainerStarted","Data":"f5ff4a0fe398120fc30656eacb3efc029e77297665e73a6998fe58e8a7f6f761"} Jul 17 00:18:19.609759 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:19.609692 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xvfn6" podStartSLOduration=66.501016906 podStartE2EDuration="1m11.609676054s" totalImagesPullingTime="5.108659148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:19.609674519 +0000 UTC m=+72.356183566" watchObservedRunningTime="2026-07-17 00:18:19.609676054 +0000 UTC m=+72.356185088" Jul 17 00:18:19.625998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:19.625953 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n78sr" podStartSLOduration=66.228014075 podStartE2EDuration="1m11.625940545s" totalImagesPullingTime="5.39792647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 00:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:18:19.624705508 +0000 UTC m=+72.371214542" watchObservedRunningTime="2026-07-17 00:18:19.625940545 +0000 UTC m=+72.372449579" Jul 17 00:18:20.250999 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:20.250967 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:20.349334 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:20.349306 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:21.317357 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:21.317324 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:21.368780 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:21.368751 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:23.580946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:23.580918 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7wrvh_30a743cd-32ba-43eb-8805-991f080b23b4/serve-healthcheck-canary/0.log" Jul 17 00:18:23.815800 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:23.815755 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:23.815800 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:23.815818 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:23.820574 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:23.820547 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:24.624513 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:24.624480 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:18:24.666054 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:24.666019 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796b6495d-gpsbp"] Jul 17 00:18:24.764392 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:24.764356 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:24.865057 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:24.865025 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:24.916229 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:24.916159 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:29.620431 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:29.620303 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:29.677163 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:29.677132 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:36.688179 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:36.688081 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:36.788569 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:36.788535 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:36.891817 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:36.891782 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:38.470696 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:38.470631 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8555f8f758-9mm99" podUID="2722bd21-96ee-446f-ba16-3f23403c297f" containerName="console" containerID="cri-o://96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369" gracePeriod=15 Jul 17 00:18:39.249174 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.249147 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:39.314623 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.314594 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-serving-cert\") pod \"2722bd21-96ee-446f-ba16-3f23403c297f\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " Jul 17 00:18:39.314771 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.314643 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-service-ca\") pod \"2722bd21-96ee-446f-ba16-3f23403c297f\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " Jul 17 00:18:39.314771 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.314672 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-oauth-serving-cert\") pod \"2722bd21-96ee-446f-ba16-3f23403c297f\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " Jul 17 00:18:39.314771 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.314695 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lfms\" (UniqueName: \"kubernetes.io/projected/2722bd21-96ee-446f-ba16-3f23403c297f-kube-api-access-9lfms\") pod \"2722bd21-96ee-446f-ba16-3f23403c297f\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " Jul 17 00:18:39.314771 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.314715 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-trusted-ca-bundle\") pod \"2722bd21-96ee-446f-ba16-3f23403c297f\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " Jul 17 00:18:39.315003 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.314785 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-oauth-config\") pod \"2722bd21-96ee-446f-ba16-3f23403c297f\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " Jul 17 00:18:39.315003 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.314832 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-console-config\") pod \"2722bd21-96ee-446f-ba16-3f23403c297f\" (UID: \"2722bd21-96ee-446f-ba16-3f23403c297f\") " Jul 17 00:18:39.315131 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.315102 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2722bd21-96ee-446f-ba16-3f23403c297f" (UID: "2722bd21-96ee-446f-ba16-3f23403c297f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:18:39.315265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.315160 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2722bd21-96ee-446f-ba16-3f23403c297f" (UID: "2722bd21-96ee-446f-ba16-3f23403c297f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:18:39.315375 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.315352 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-console-config" (OuterVolumeSpecName: "console-config") pod "2722bd21-96ee-446f-ba16-3f23403c297f" (UID: "2722bd21-96ee-446f-ba16-3f23403c297f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:18:39.315431 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.315356 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-service-ca" (OuterVolumeSpecName: "service-ca") pod "2722bd21-96ee-446f-ba16-3f23403c297f" (UID: "2722bd21-96ee-446f-ba16-3f23403c297f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:18:39.317000 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.316979 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2722bd21-96ee-446f-ba16-3f23403c297f-kube-api-access-9lfms" (OuterVolumeSpecName: "kube-api-access-9lfms") pod "2722bd21-96ee-446f-ba16-3f23403c297f" (UID: "2722bd21-96ee-446f-ba16-3f23403c297f"). InnerVolumeSpecName "kube-api-access-9lfms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 00:18:39.317217 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.317197 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2722bd21-96ee-446f-ba16-3f23403c297f" (UID: "2722bd21-96ee-446f-ba16-3f23403c297f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 00:18:39.317284 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.317259 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2722bd21-96ee-446f-ba16-3f23403c297f" (UID: "2722bd21-96ee-446f-ba16-3f23403c297f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 00:18:39.416265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.416183 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:39.416265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.416212 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-oauth-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:39.416265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.416222 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lfms\" (UniqueName: \"kubernetes.io/projected/2722bd21-96ee-446f-ba16-3f23403c297f-kube-api-access-9lfms\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:39.416265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.416231 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-trusted-ca-bundle\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:39.416265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.416241 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-oauth-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:39.416265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.416249 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2722bd21-96ee-446f-ba16-3f23403c297f-console-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:39.416265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.416258 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2722bd21-96ee-446f-ba16-3f23403c297f-console-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:39.685919 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.685820 2572 generic.go:356] "Generic (PLEG): container finished" podID="2722bd21-96ee-446f-ba16-3f23403c297f" containerID="96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369" exitCode=0 Jul 17 00:18:39.685919 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.685867 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8555f8f758-9mm99" Jul 17 00:18:39.686326 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.685915 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8555f8f758-9mm99" event={"ID":"2722bd21-96ee-446f-ba16-3f23403c297f","Type":"ContainerDied","Data":"96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369"} Jul 17 00:18:39.686326 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.685968 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8555f8f758-9mm99" event={"ID":"2722bd21-96ee-446f-ba16-3f23403c297f","Type":"ContainerDied","Data":"907e611ff4760607a5831aa391279591059c6f35860c24082a926609dd1e6444"} Jul 17 00:18:39.686326 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.685988 2572 scope.go:117] "RemoveContainer" containerID="96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369" Jul 17 00:18:39.694683 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.694668 2572 scope.go:117] "RemoveContainer" containerID="96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369" Jul 17 00:18:39.694913 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:18:39.694881 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369\": container with ID starting with 96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369 not found: ID does not exist" containerID="96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369" Jul 17 00:18:39.694968 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.694920 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369"} err="failed to get container status \"96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369\": rpc error: code = NotFound desc = could not find container \"96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369\": container with ID starting with 96ef615db76a99ce7236c1111b3feb08d7b0cf68c9864b0fe7ab6786b8ac7369 not found: ID does not exist" Jul 17 00:18:39.705550 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.705521 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8555f8f758-9mm99"] Jul 17 00:18:39.707502 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:39.707481 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8555f8f758-9mm99"] Jul 17 00:18:40.087309 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:40.087237 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2722bd21-96ee-446f-ba16-3f23403c297f" path="/var/lib/kubelet/pods/2722bd21-96ee-446f-ba16-3f23403c297f/volumes" Jul 17 00:18:44.594337 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:44.594300 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:44.660201 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:44.660174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:18:49.688032 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:49.687986 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-796b6495d-gpsbp" podUID="92b088cb-58ec-48be-8fbb-7e28edbbda10" containerName="console" containerID="cri-o://83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d" gracePeriod=15 Jul 17 00:18:49.951855 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:49.951788 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:18:50.110922 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.110871 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-serving-cert\") pod \"92b088cb-58ec-48be-8fbb-7e28edbbda10\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " Jul 17 00:18:50.110922 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.110923 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-oauth-serving-cert\") pod \"92b088cb-58ec-48be-8fbb-7e28edbbda10\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " Jul 17 00:18:50.111140 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111039 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-service-ca\") pod \"92b088cb-58ec-48be-8fbb-7e28edbbda10\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " Jul 17 00:18:50.111140 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111078 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-oauth-config\") pod \"92b088cb-58ec-48be-8fbb-7e28edbbda10\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " Jul 17 00:18:50.111140 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111106 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-trusted-ca-bundle\") pod \"92b088cb-58ec-48be-8fbb-7e28edbbda10\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " Jul 17 00:18:50.111292 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111244 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-config\") pod \"92b088cb-58ec-48be-8fbb-7e28edbbda10\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " Jul 17 00:18:50.111353 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111302 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjllx\" (UniqueName: \"kubernetes.io/projected/92b088cb-58ec-48be-8fbb-7e28edbbda10-kube-api-access-rjllx\") pod \"92b088cb-58ec-48be-8fbb-7e28edbbda10\" (UID: \"92b088cb-58ec-48be-8fbb-7e28edbbda10\") " Jul 17 00:18:50.111407 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111353 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "92b088cb-58ec-48be-8fbb-7e28edbbda10" (UID: "92b088cb-58ec-48be-8fbb-7e28edbbda10"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:18:50.111460 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111443 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "92b088cb-58ec-48be-8fbb-7e28edbbda10" (UID: "92b088cb-58ec-48be-8fbb-7e28edbbda10"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:18:50.111561 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111455 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-service-ca" (OuterVolumeSpecName: "service-ca") pod "92b088cb-58ec-48be-8fbb-7e28edbbda10" (UID: "92b088cb-58ec-48be-8fbb-7e28edbbda10"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:18:50.111676 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111655 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:50.111676 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111678 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-trusted-ca-bundle\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:50.111828 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111692 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-oauth-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:50.111828 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.111647 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-config" (OuterVolumeSpecName: "console-config") pod "92b088cb-58ec-48be-8fbb-7e28edbbda10" (UID: "92b088cb-58ec-48be-8fbb-7e28edbbda10"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:18:50.113268 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.113239 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "92b088cb-58ec-48be-8fbb-7e28edbbda10" (UID: "92b088cb-58ec-48be-8fbb-7e28edbbda10"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 00:18:50.113397 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.113375 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "92b088cb-58ec-48be-8fbb-7e28edbbda10" (UID: "92b088cb-58ec-48be-8fbb-7e28edbbda10"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 00:18:50.113397 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.113386 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b088cb-58ec-48be-8fbb-7e28edbbda10-kube-api-access-rjllx" (OuterVolumeSpecName: "kube-api-access-rjllx") pod "92b088cb-58ec-48be-8fbb-7e28edbbda10" (UID: "92b088cb-58ec-48be-8fbb-7e28edbbda10"). InnerVolumeSpecName "kube-api-access-rjllx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 00:18:50.212383 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.212297 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:50.212383 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.212323 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-oauth-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:50.212556 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.212402 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b088cb-58ec-48be-8fbb-7e28edbbda10-console-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:50.212556 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.212412 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjllx\" (UniqueName: \"kubernetes.io/projected/92b088cb-58ec-48be-8fbb-7e28edbbda10-kube-api-access-rjllx\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:18:50.605091 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.605067 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xvfn6" Jul 17 00:18:50.733220 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.733185 2572 generic.go:356] "Generic (PLEG): container finished" podID="92b088cb-58ec-48be-8fbb-7e28edbbda10" containerID="83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d" exitCode=0 Jul 17 00:18:50.733589 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.733230 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796b6495d-gpsbp" event={"ID":"92b088cb-58ec-48be-8fbb-7e28edbbda10","Type":"ContainerDied","Data":"83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d"} Jul 17 00:18:50.733589 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.733270 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796b6495d-gpsbp" event={"ID":"92b088cb-58ec-48be-8fbb-7e28edbbda10","Type":"ContainerDied","Data":"65e53c380dfe8f12ed7ad387c19d2a34e9e33c529daee40d24c727886458e1af"} Jul 17 00:18:50.733589 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.733289 2572 scope.go:117] "RemoveContainer" containerID="83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d" Jul 17 00:18:50.733589 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.733234 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796b6495d-gpsbp" Jul 17 00:18:50.742015 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.741996 2572 scope.go:117] "RemoveContainer" containerID="83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d" Jul 17 00:18:50.742265 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:18:50.742245 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d\": container with ID starting with 83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d not found: ID does not exist" containerID="83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d" Jul 17 00:18:50.742329 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.742277 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d"} err="failed to get container status \"83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d\": rpc error: code = NotFound desc = could not find container \"83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d\": container with ID starting with 83fc377ff32d9e0561cf6c4f6b370ca423a1aabc3a399db61da4fe19f865f18d not found: ID does not exist" Jul 17 00:18:50.752710 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.752689 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796b6495d-gpsbp"] Jul 17 00:18:50.758348 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:50.758331 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-796b6495d-gpsbp"] Jul 17 00:18:52.086850 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:18:52.086817 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b088cb-58ec-48be-8fbb-7e28edbbda10" path="/var/lib/kubelet/pods/92b088cb-58ec-48be-8fbb-7e28edbbda10/volumes" Jul 17 00:19:10.213048 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:10.213015 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:19:10.233297 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:10.233273 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:19:10.825003 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:10.824975 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 00:19:19.716054 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:19.716020 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:19.771322 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:19.771290 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:31.479180 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:31.479145 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:31.625201 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:31.625168 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:31.677566 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:31.677532 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:31.739764 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:31.739702 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:31.791551 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:31.791528 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:32.224858 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:32.224830 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:32.326143 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:32.326111 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:42.746715 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:42.746683 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866b45cb98-28qc2"] Jul 17 00:19:42.846879 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:42.846826 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:42.977973 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:42.977939 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:43.038067 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:43.037986 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:44.605510 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:44.605478 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:44.703730 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:44.703698 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:19:53.732294 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:19:53.732260 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:04.063145 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:04.063076 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:04.119832 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:04.119806 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:07.767386 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:07.767351 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-866b45cb98-28qc2" podUID="c5234caf-2e88-4d8e-bd8f-fee075a254d8" containerName="console" containerID="cri-o://3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751" gracePeriod=15 Jul 17 00:20:08.017009 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.016986 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:20:08.026302 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.026245 2572 generic.go:356] "Generic (PLEG): container finished" podID="c5234caf-2e88-4d8e-bd8f-fee075a254d8" containerID="3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751" exitCode=0 Jul 17 00:20:08.026302 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.026280 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866b45cb98-28qc2" Jul 17 00:20:08.026441 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.026318 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866b45cb98-28qc2" event={"ID":"c5234caf-2e88-4d8e-bd8f-fee075a254d8","Type":"ContainerDied","Data":"3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751"} Jul 17 00:20:08.026441 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.026366 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866b45cb98-28qc2" event={"ID":"c5234caf-2e88-4d8e-bd8f-fee075a254d8","Type":"ContainerDied","Data":"a5126ab3296ed49febfaf66465f8047f278e4ccad0db1ced76f162eb4f045446"} Jul 17 00:20:08.026441 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.026387 2572 scope.go:117] "RemoveContainer" containerID="3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751" Jul 17 00:20:08.034238 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.034219 2572 scope.go:117] "RemoveContainer" containerID="3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751" Jul 17 00:20:08.034873 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:20:08.034660 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751\": container with ID starting with 3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751 not found: ID does not exist" containerID="3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751" Jul 17 00:20:08.034873 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.034716 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751"} err="failed to get container status \"3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751\": rpc error: code = NotFound desc = could not find container \"3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751\": container with ID starting with 3e63d47829a2939aea1dd2f8c45bee4de8a28a4772a7b859c0bdc1ebba418751 not found: ID does not exist" Jul 17 00:20:08.063805 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.063782 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-config\") pod \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " Jul 17 00:20:08.063909 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.063813 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-serving-cert\") pod \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " Jul 17 00:20:08.063909 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.063829 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-oauth-serving-cert\") pod \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " Jul 17 00:20:08.063909 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.063861 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-service-ca\") pod \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " Jul 17 00:20:08.064028 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.063910 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-oauth-config\") pod \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " Jul 17 00:20:08.064028 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.063954 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnjbp\" (UniqueName: \"kubernetes.io/projected/c5234caf-2e88-4d8e-bd8f-fee075a254d8-kube-api-access-pnjbp\") pod \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " Jul 17 00:20:08.064028 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.064012 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-trusted-ca-bundle\") pod \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\" (UID: \"c5234caf-2e88-4d8e-bd8f-fee075a254d8\") " Jul 17 00:20:08.064272 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.064227 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-config" (OuterVolumeSpecName: "console-config") pod "c5234caf-2e88-4d8e-bd8f-fee075a254d8" (UID: "c5234caf-2e88-4d8e-bd8f-fee075a254d8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:20:08.064474 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.064286 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-service-ca" (OuterVolumeSpecName: "service-ca") pod "c5234caf-2e88-4d8e-bd8f-fee075a254d8" (UID: "c5234caf-2e88-4d8e-bd8f-fee075a254d8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:20:08.064474 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.064304 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:20:08.064474 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.064406 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c5234caf-2e88-4d8e-bd8f-fee075a254d8" (UID: "c5234caf-2e88-4d8e-bd8f-fee075a254d8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:20:08.064617 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.064491 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c5234caf-2e88-4d8e-bd8f-fee075a254d8" (UID: "c5234caf-2e88-4d8e-bd8f-fee075a254d8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 00:20:08.066092 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.066070 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c5234caf-2e88-4d8e-bd8f-fee075a254d8" (UID: "c5234caf-2e88-4d8e-bd8f-fee075a254d8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 00:20:08.066187 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.066141 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c5234caf-2e88-4d8e-bd8f-fee075a254d8" (UID: "c5234caf-2e88-4d8e-bd8f-fee075a254d8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 00:20:08.066187 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.066152 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5234caf-2e88-4d8e-bd8f-fee075a254d8-kube-api-access-pnjbp" (OuterVolumeSpecName: "kube-api-access-pnjbp") pod "c5234caf-2e88-4d8e-bd8f-fee075a254d8" (UID: "c5234caf-2e88-4d8e-bd8f-fee075a254d8"). InnerVolumeSpecName "kube-api-access-pnjbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 00:20:08.165609 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.165587 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:20:08.165609 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.165608 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-oauth-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:20:08.165722 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.165617 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:20:08.165722 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.165626 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5234caf-2e88-4d8e-bd8f-fee075a254d8-console-oauth-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:20:08.165722 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.165635 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnjbp\" (UniqueName: \"kubernetes.io/projected/c5234caf-2e88-4d8e-bd8f-fee075a254d8-kube-api-access-pnjbp\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:20:08.165722 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.165644 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5234caf-2e88-4d8e-bd8f-fee075a254d8-trusted-ca-bundle\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:20:08.340834 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.340779 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866b45cb98-28qc2"] Jul 17 00:20:08.344387 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:08.344366 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-866b45cb98-28qc2"] Jul 17 00:20:10.086139 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:10.086102 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5234caf-2e88-4d8e-bd8f-fee075a254d8" path="/var/lib/kubelet/pods/c5234caf-2e88-4d8e-bd8f-fee075a254d8/volumes" Jul 17 00:20:16.483412 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:16.483375 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:16.578834 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:16.578803 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:43.068680 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:43.068648 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:43.435563 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:43.435534 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:44.909150 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:44.909116 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:47.066589 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:47.066560 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:47.167129 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:47.167091 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:48.373305 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:48.373273 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:48.895746 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:48.895714 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:48.947779 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:48.947754 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:50.841301 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:50.841275 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:51.258854 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:51.258827 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:53.705504 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705472 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8"] Jul 17 00:20:53.705912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705795 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5234caf-2e88-4d8e-bd8f-fee075a254d8" containerName="console" Jul 17 00:20:53.705912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705806 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5234caf-2e88-4d8e-bd8f-fee075a254d8" containerName="console" Jul 17 00:20:53.705912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705816 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b088cb-58ec-48be-8fbb-7e28edbbda10" containerName="console" Jul 17 00:20:53.705912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705822 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b088cb-58ec-48be-8fbb-7e28edbbda10" containerName="console" Jul 17 00:20:53.705912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705833 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2722bd21-96ee-446f-ba16-3f23403c297f" containerName="console" Jul 17 00:20:53.705912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705839 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2722bd21-96ee-446f-ba16-3f23403c297f" containerName="console" Jul 17 00:20:53.705912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705899 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b088cb-58ec-48be-8fbb-7e28edbbda10" containerName="console" Jul 17 00:20:53.705912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705909 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5234caf-2e88-4d8e-bd8f-fee075a254d8" containerName="console" Jul 17 00:20:53.705912 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.705916 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2722bd21-96ee-446f-ba16-3f23403c297f" containerName="console" Jul 17 00:20:53.708823 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.708805 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" Jul 17 00:20:53.711167 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.711148 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 00:20:53.711263 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.711152 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tr62f\"" Jul 17 00:20:53.712101 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.712086 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 00:20:53.718215 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.718194 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8"] Jul 17 00:20:53.806819 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.806795 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5fffac3-7924-489e-91ed-2d880dd3232e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-sgrt8\" (UID: \"a5fffac3-7924-489e-91ed-2d880dd3232e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" Jul 17 00:20:53.806938 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.806850 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndlc\" (UniqueName: \"kubernetes.io/projected/a5fffac3-7924-489e-91ed-2d880dd3232e-kube-api-access-4ndlc\") pod \"cert-manager-cainjector-559cdd8c87-sgrt8\" (UID: \"a5fffac3-7924-489e-91ed-2d880dd3232e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" Jul 17 00:20:53.857565 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.857540 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:53.907712 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.907686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5fffac3-7924-489e-91ed-2d880dd3232e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-sgrt8\" (UID: \"a5fffac3-7924-489e-91ed-2d880dd3232e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" Jul 17 00:20:53.907810 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.907771 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndlc\" (UniqueName: \"kubernetes.io/projected/a5fffac3-7924-489e-91ed-2d880dd3232e-kube-api-access-4ndlc\") pod \"cert-manager-cainjector-559cdd8c87-sgrt8\" (UID: \"a5fffac3-7924-489e-91ed-2d880dd3232e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" Jul 17 00:20:53.914612 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.914589 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5fffac3-7924-489e-91ed-2d880dd3232e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-sgrt8\" (UID: \"a5fffac3-7924-489e-91ed-2d880dd3232e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" Jul 17 00:20:53.917833 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.917814 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndlc\" (UniqueName: \"kubernetes.io/projected/a5fffac3-7924-489e-91ed-2d880dd3232e-kube-api-access-4ndlc\") pod \"cert-manager-cainjector-559cdd8c87-sgrt8\" (UID: \"a5fffac3-7924-489e-91ed-2d880dd3232e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" Jul 17 00:20:53.958084 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:53.958033 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:54.024743 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:54.024721 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" Jul 17 00:20:54.145700 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:54.145679 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8"] Jul 17 00:20:54.148245 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:20:54.148217 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5fffac3_7924_489e_91ed_2d880dd3232e.slice/crio-33e64a4d29601b11ba1a065ef26a6365bd4ea1a1c8c5e3909facbaaab17eeaef WatchSource:0}: Error finding container 33e64a4d29601b11ba1a065ef26a6365bd4ea1a1c8c5e3909facbaaab17eeaef: Status 404 returned error can't find the container with id 33e64a4d29601b11ba1a065ef26a6365bd4ea1a1c8c5e3909facbaaab17eeaef Jul 17 00:20:54.197315 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:54.197288 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" event={"ID":"a5fffac3-7924-489e-91ed-2d880dd3232e","Type":"ContainerStarted","Data":"33e64a4d29601b11ba1a065ef26a6365bd4ea1a1c8c5e3909facbaaab17eeaef"} Jul 17 00:20:58.044786 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:58.044755 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:20:58.214904 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:58.214854 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" event={"ID":"a5fffac3-7924-489e-91ed-2d880dd3232e","Type":"ContainerStarted","Data":"9f279fade12acc693b62d4fb9d1cf8e70c81be157369404dea1481431ae20192"} Jul 17 00:20:58.231668 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:20:58.231619 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sgrt8" podStartSLOduration=1.570864383 podStartE2EDuration="5.231603848s" totalImagesPullingTime="3.660739465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:20:58.230961496 +0000 UTC m=+230.977470542" watchObservedRunningTime="2026-07-17 00:20:58.231603848 +0000 UTC m=+230.978112881" Jul 17 00:21:00.254779 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:00.254751 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:00.329769 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:00.329745 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:02.270582 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.270551 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz"] Jul 17 00:21:02.274075 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.274055 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" Jul 17 00:21:02.275958 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.275933 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a7d1aea-d729-4cb5-9f9a-04d40eadeed7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pkpfz\" (UID: \"8a7d1aea-d729-4cb5-9f9a-04d40eadeed7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" Jul 17 00:21:02.276081 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.275997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgt5\" (UniqueName: \"kubernetes.io/projected/8a7d1aea-d729-4cb5-9f9a-04d40eadeed7-kube-api-access-wcgt5\") pod \"openshift-lws-operator-fd8ccff4c-pkpfz\" (UID: \"8a7d1aea-d729-4cb5-9f9a-04d40eadeed7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" Jul 17 00:21:02.276658 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.276640 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 00:21:02.277639 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.277625 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-p55ck\"" Jul 17 00:21:02.277692 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.277635 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 00:21:02.283233 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.283213 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz"] Jul 17 00:21:02.376788 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.376763 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a7d1aea-d729-4cb5-9f9a-04d40eadeed7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pkpfz\" (UID: \"8a7d1aea-d729-4cb5-9f9a-04d40eadeed7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" Jul 17 00:21:02.377080 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.376812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgt5\" (UniqueName: \"kubernetes.io/projected/8a7d1aea-d729-4cb5-9f9a-04d40eadeed7-kube-api-access-wcgt5\") pod \"openshift-lws-operator-fd8ccff4c-pkpfz\" (UID: \"8a7d1aea-d729-4cb5-9f9a-04d40eadeed7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" Jul 17 00:21:02.377142 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.377114 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a7d1aea-d729-4cb5-9f9a-04d40eadeed7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pkpfz\" (UID: \"8a7d1aea-d729-4cb5-9f9a-04d40eadeed7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" Jul 17 00:21:02.384501 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.384480 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgt5\" (UniqueName: \"kubernetes.io/projected/8a7d1aea-d729-4cb5-9f9a-04d40eadeed7-kube-api-access-wcgt5\") pod \"openshift-lws-operator-fd8ccff4c-pkpfz\" (UID: \"8a7d1aea-d729-4cb5-9f9a-04d40eadeed7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" Jul 17 00:21:02.583089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.583025 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" Jul 17 00:21:02.704409 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:02.704388 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz"] Jul 17 00:21:02.706820 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:21:02.706787 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a7d1aea_d729_4cb5_9f9a_04d40eadeed7.slice/crio-2c5ed8e8cb9463fe865d45e88fca8ef17311ce2fe99712cbb6643e6378a7d465 WatchSource:0}: Error finding container 2c5ed8e8cb9463fe865d45e88fca8ef17311ce2fe99712cbb6643e6378a7d465: Status 404 returned error can't find the container with id 2c5ed8e8cb9463fe865d45e88fca8ef17311ce2fe99712cbb6643e6378a7d465 Jul 17 00:21:03.234770 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:03.234735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" event={"ID":"8a7d1aea-d729-4cb5-9f9a-04d40eadeed7","Type":"ContainerStarted","Data":"2c5ed8e8cb9463fe865d45e88fca8ef17311ce2fe99712cbb6643e6378a7d465"} Jul 17 00:21:03.509229 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:03.509143 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:03.611596 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:03.611565 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:06.249383 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:06.249345 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" event={"ID":"8a7d1aea-d729-4cb5-9f9a-04d40eadeed7","Type":"ContainerStarted","Data":"315176736ed054017b7e9957f3324736eda9e2c7ea3d526745807cc0e59e9bcf"} Jul 17 00:21:06.478453 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:06.478420 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:06.579442 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:06.579365 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:06.960472 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:06.960441 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:07.061265 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:07.061239 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:09.158310 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:09.158281 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:21.955510 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:21.955476 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:22.054913 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:22.054864 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:22.505662 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:22.505629 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:23.068599 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.068562 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:23.158175 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.158146 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:23.185294 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.185247 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pkpfz" podStartSLOduration=18.621360583 podStartE2EDuration="21.185232553s" totalImagesPullingTime="2.56387197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:06.265673114 +0000 UTC m=+239.012182163" watchObservedRunningTime="2026-07-17 00:21:23.185232553 +0000 UTC m=+255.931741589" Jul 17 00:21:23.185480 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.185466 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9"] Jul 17 00:21:23.190141 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.190125 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.193386 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.193364 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 00:21:23.193477 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.193464 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 00:21:23.193749 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.193733 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g6jft\"" Jul 17 00:21:23.193806 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.193733 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 00:21:23.194113 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.194099 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 00:21:23.204214 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.204185 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9"] Jul 17 00:21:23.236227 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.236200 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmg8\" (UniqueName: \"kubernetes.io/projected/c2725a15-6cb6-4252-aa67-52c3aac340f0-kube-api-access-4cmg8\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.236322 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.236246 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2725a15-6cb6-4252-aa67-52c3aac340f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.236322 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.236276 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2725a15-6cb6-4252-aa67-52c3aac340f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.236392 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.236348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2725a15-6cb6-4252-aa67-52c3aac340f0-tmp\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.236425 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.236408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2725a15-6cb6-4252-aa67-52c3aac340f0-manifests\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.301797 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.301756 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:23.337454 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.337406 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2725a15-6cb6-4252-aa67-52c3aac340f0-manifests\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.337454 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.337443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmg8\" (UniqueName: \"kubernetes.io/projected/c2725a15-6cb6-4252-aa67-52c3aac340f0-kube-api-access-4cmg8\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.337580 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.337464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2725a15-6cb6-4252-aa67-52c3aac340f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.337580 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.337490 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2725a15-6cb6-4252-aa67-52c3aac340f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.337580 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.337544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2725a15-6cb6-4252-aa67-52c3aac340f0-tmp\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.337794 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.337775 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2725a15-6cb6-4252-aa67-52c3aac340f0-manifests\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.339669 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.339642 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2725a15-6cb6-4252-aa67-52c3aac340f0-tmp\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.339848 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.339833 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2725a15-6cb6-4252-aa67-52c3aac340f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.339916 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.339844 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2725a15-6cb6-4252-aa67-52c3aac340f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.347678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.347659 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmg8\" (UniqueName: \"kubernetes.io/projected/c2725a15-6cb6-4252-aa67-52c3aac340f0-kube-api-access-4cmg8\") pod \"opendatahub-operator-controller-manager-c9c65dfff-nswv9\" (UID: \"c2725a15-6cb6-4252-aa67-52c3aac340f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.448413 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.448387 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:23.500470 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.500437 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:23.550432 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.550396 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:23.630328 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.630298 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9"] Jul 17 00:21:23.632803 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:21:23.632779 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2725a15_6cb6_4252_aa67_52c3aac340f0.slice/crio-ad11a8a07fad96cbe6174657f1eb1898f9ef5e55f11a42f015f5e4bb65c60e1f WatchSource:0}: Error finding container ad11a8a07fad96cbe6174657f1eb1898f9ef5e55f11a42f015f5e4bb65c60e1f: Status 404 returned error can't find the container with id ad11a8a07fad96cbe6174657f1eb1898f9ef5e55f11a42f015f5e4bb65c60e1f Jul 17 00:21:23.637761 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.637738 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:23.774580 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:23.774548 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:24.320793 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:24.320759 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" event={"ID":"c2725a15-6cb6-4252-aa67-52c3aac340f0","Type":"ContainerStarted","Data":"ad11a8a07fad96cbe6174657f1eb1898f9ef5e55f11a42f015f5e4bb65c60e1f"} Jul 17 00:21:26.983188 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:26.983125 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:27.127540 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:27.127515 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:27.228765 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:27.228738 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:27.336205 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:27.336152 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" event={"ID":"c2725a15-6cb6-4252-aa67-52c3aac340f0","Type":"ContainerStarted","Data":"8aed0d2a42f81cbcee5d9aa7055e6b51b2d3262997be042956e932440a2c0ee9"} Jul 17 00:21:28.341348 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:28.341318 2572 generic.go:356] "Generic (PLEG): container finished" podID="c2725a15-6cb6-4252-aa67-52c3aac340f0" containerID="8aed0d2a42f81cbcee5d9aa7055e6b51b2d3262997be042956e932440a2c0ee9" exitCode=0 Jul 17 00:21:28.341348 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:28.341349 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" event={"ID":"c2725a15-6cb6-4252-aa67-52c3aac340f0","Type":"ContainerDied","Data":"8aed0d2a42f81cbcee5d9aa7055e6b51b2d3262997be042956e932440a2c0ee9"} Jul 17 00:21:29.347646 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:29.347614 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" event={"ID":"c2725a15-6cb6-4252-aa67-52c3aac340f0","Type":"ContainerStarted","Data":"bb03ce18cff6333c9c3ff83edd1c1a42c761ac7f670608f4ea44d6b9711c8006"} Jul 17 00:21:29.348030 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:29.347682 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:29.373393 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:29.373348 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" podStartSLOduration=1.823254956 podStartE2EDuration="6.373335608s" totalImagesPullingTime="3.561766192s" totalInitContainerRuntime="988.31446ms" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:29.372000337 +0000 UTC m=+262.118509369" watchObservedRunningTime="2026-07-17 00:21:29.373335608 +0000 UTC m=+262.119844641" Jul 17 00:21:34.858689 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:34.858660 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:34.947240 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:34.947156 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:40.354040 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:40.354005 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c9c65dfff-nswv9" Jul 17 00:21:41.850188 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:41.850134 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:42.005157 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.005129 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:42.318508 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.318424 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:42.587717 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.587629 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d679d599d-prk4d"] Jul 17 00:21:42.599027 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.598997 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.604979 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.604950 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-68ctg\"" Jul 17 00:21:42.605413 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.605396 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 00:21:42.605753 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.605738 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 00:21:42.606064 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.606049 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 00:21:42.606438 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.606424 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 00:21:42.609814 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.609792 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d679d599d-prk4d"] Jul 17 00:21:42.612306 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.612284 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:42.689061 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.689032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d06e5708-5361-4e4c-88e2-2525d9001cf3-tmp\") pod \"kube-auth-proxy-d679d599d-prk4d\" (UID: \"d06e5708-5361-4e4c-88e2-2525d9001cf3\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.689241 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.689095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06e5708-5361-4e4c-88e2-2525d9001cf3-tls-certs\") pod \"kube-auth-proxy-d679d599d-prk4d\" (UID: \"d06e5708-5361-4e4c-88e2-2525d9001cf3\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.689241 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.689125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qm68\" (UniqueName: \"kubernetes.io/projected/d06e5708-5361-4e4c-88e2-2525d9001cf3-kube-api-access-5qm68\") pod \"kube-auth-proxy-d679d599d-prk4d\" (UID: \"d06e5708-5361-4e4c-88e2-2525d9001cf3\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.757183 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.757149 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:42.790118 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.790090 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06e5708-5361-4e4c-88e2-2525d9001cf3-tls-certs\") pod \"kube-auth-proxy-d679d599d-prk4d\" (UID: \"d06e5708-5361-4e4c-88e2-2525d9001cf3\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.790298 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.790132 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qm68\" (UniqueName: \"kubernetes.io/projected/d06e5708-5361-4e4c-88e2-2525d9001cf3-kube-api-access-5qm68\") pod \"kube-auth-proxy-d679d599d-prk4d\" (UID: \"d06e5708-5361-4e4c-88e2-2525d9001cf3\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.790298 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.790195 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d06e5708-5361-4e4c-88e2-2525d9001cf3-tmp\") pod \"kube-auth-proxy-d679d599d-prk4d\" (UID: \"d06e5708-5361-4e4c-88e2-2525d9001cf3\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.792406 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.792380 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d06e5708-5361-4e4c-88e2-2525d9001cf3-tmp\") pod \"kube-auth-proxy-d679d599d-prk4d\" (UID: \"d06e5708-5361-4e4c-88e2-2525d9001cf3\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.792711 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.792693 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06e5708-5361-4e4c-88e2-2525d9001cf3-tls-certs\") pod \"kube-auth-proxy-d679d599d-prk4d\" (UID: \"d06e5708-5361-4e4c-88e2-2525d9001cf3\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.827103 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.827070 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qm68\" (UniqueName: \"kubernetes.io/projected/d06e5708-5361-4e4c-88e2-2525d9001cf3-kube-api-access-5qm68\") pod \"kube-auth-proxy-d679d599d-prk4d\" (UID: \"d06e5708-5361-4e4c-88e2-2525d9001cf3\") " pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:42.899004 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.898970 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:42.916073 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:42.916049 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" Jul 17 00:21:43.040671 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:43.040588 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d679d599d-prk4d"] Jul 17 00:21:43.043258 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:21:43.043232 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06e5708_5361_4e4c_88e2_2525d9001cf3.slice/crio-cb84671898496d5985a034e1e218841a84663bad99d33b7f0c9c501429a96921 WatchSource:0}: Error finding container cb84671898496d5985a034e1e218841a84663bad99d33b7f0c9c501429a96921: Status 404 returned error can't find the container with id cb84671898496d5985a034e1e218841a84663bad99d33b7f0c9c501429a96921 Jul 17 00:21:43.060868 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:43.060839 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:43.407937 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:43.407909 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" event={"ID":"d06e5708-5361-4e4c-88e2-2525d9001cf3","Type":"ContainerStarted","Data":"cb84671898496d5985a034e1e218841a84663bad99d33b7f0c9c501429a96921"} Jul 17 00:21:43.478212 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:43.478160 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:43.620527 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:43.620493 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:44.102156 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:44.102119 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:44.168009 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:44.167968 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:45.158296 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:45.158262 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:46.423476 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:46.423438 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" event={"ID":"d06e5708-5361-4e4c-88e2-2525d9001cf3","Type":"ContainerStarted","Data":"3344f732aaf196209ce6a92a42f92d26dddeb1efc17f0eaacdbdd4c6b4d406a3"} Jul 17 00:21:46.441461 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:46.441415 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d679d599d-prk4d" podStartSLOduration=1.712472172 podStartE2EDuration="4.441400007s" totalImagesPullingTime="2.728927835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:46.439129766 +0000 UTC m=+279.185638820" watchObservedRunningTime="2026-07-17 00:21:46.441400007 +0000 UTC m=+279.187909049" Jul 17 00:21:50.781646 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.781613 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt"] Jul 17 00:21:50.785550 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.785518 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.789958 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.789937 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 00:21:50.791013 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.790990 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 00:21:50.791126 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.791086 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 00:21:50.791126 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.791033 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rtttq\"" Jul 17 00:21:50.793117 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.793097 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt"] Jul 17 00:21:50.863143 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.863111 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-metrics-cert\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.863301 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.863145 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wjl8\" (UniqueName: \"kubernetes.io/projected/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-kube-api-access-6wjl8\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.863301 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.863249 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-manager-config\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.863301 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.863281 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-cert\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.964620 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.964585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-metrics-cert\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.964620 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.964625 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wjl8\" (UniqueName: \"kubernetes.io/projected/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-kube-api-access-6wjl8\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.964862 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.964659 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-manager-config\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.964862 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.964686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-cert\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.965451 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.965426 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-manager-config\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.967247 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.967225 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-cert\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.967330 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.967282 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-metrics-cert\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:50.979334 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:50.979313 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wjl8\" (UniqueName: \"kubernetes.io/projected/443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5-kube-api-access-6wjl8\") pod \"lws-controller-manager-6d74cbdbc7-jt9zt\" (UID: \"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5\") " pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:51.098157 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:51.098133 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:51.226749 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:51.226722 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt"] Jul 17 00:21:51.229452 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:21:51.229422 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443bb4e3_bb3b_4c05_a5a9_17559ebaf4e5.slice/crio-3cae0e7f12944fde721148bbf5bd62621221b10e29fd778a70687e8e615f7cf5 WatchSource:0}: Error finding container 3cae0e7f12944fde721148bbf5bd62621221b10e29fd778a70687e8e615f7cf5: Status 404 returned error can't find the container with id 3cae0e7f12944fde721148bbf5bd62621221b10e29fd778a70687e8e615f7cf5 Jul 17 00:21:51.445151 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:51.445064 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" event={"ID":"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5","Type":"ContainerStarted","Data":"3cae0e7f12944fde721148bbf5bd62621221b10e29fd778a70687e8e615f7cf5"} Jul 17 00:21:53.456538 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:53.456506 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" event={"ID":"443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5","Type":"ContainerStarted","Data":"6d21146bdf8699f49bae9c24eaf1b8996d7ea83b5990aa181453cbea7c5ee07f"} Jul 17 00:21:53.456930 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:53.456595 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:21:53.477523 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:53.477453 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" podStartSLOduration=1.904984749 podStartE2EDuration="3.477431534s" totalImagesPullingTime="1.572446785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:21:53.47615114 +0000 UTC m=+286.222660176" watchObservedRunningTime="2026-07-17 00:21:53.477431534 +0000 UTC m=+286.223940569" Jul 17 00:21:53.490175 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:53.490140 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:53.603608 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:53.603577 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:53.876215 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:53.876187 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:54.265296 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:54.265216 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:54.674348 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:54.674308 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:54.768951 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:54.768922 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:54.867473 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:54.867439 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:54.967904 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:54.967821 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:55.503389 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:55.503360 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:55.783216 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:55.783139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:21:55.879757 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:21:55.879728 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:22:04.462640 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:22:04.462608 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d74cbdbc7-jt9zt" Jul 17 00:22:07.722282 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:22:07.722250 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 00:22:43.321546 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:22:43.321512 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:22:43.420899 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:22:43.420859 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:22:43.493253 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:22:43.493223 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:22:52.436342 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:22:52.436307 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:22:52.489246 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:22:52.489216 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:22:54.396269 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:22:54.396236 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:22:54.493646 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:22:54.493620 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:12.333608 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:12.333529 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:13.051905 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:13.051856 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:13.962242 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:13.962207 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:14.082701 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:14.082672 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:17.351834 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:17.351805 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:17.750369 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:17.750298 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:17.873427 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:17.873401 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:18.104020 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:18.103988 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:18.185446 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:18.185419 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:19.971808 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:19.971774 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:20.194751 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:20.194722 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:21.199221 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:21.199190 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:23.946633 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:23.946604 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:24.984023 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:24.983991 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:25.671675 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.671648 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn"] Jul 17 00:23:25.674918 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.674881 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:25.679273 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.679249 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 00:23:25.679374 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.679327 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-k6d6s\"" Jul 17 00:23:25.679374 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.679327 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 00:23:25.689324 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.689299 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn"] Jul 17 00:23:25.789953 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.789928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnd7k\" (UniqueName: \"kubernetes.io/projected/8b3a80ba-8916-4ce4-8a8b-401b8d6fe573-kube-api-access-bnd7k\") pod \"limitador-operator-controller-manager-859d7b78bd-8btrn\" (UID: \"8b3a80ba-8916-4ce4-8a8b-401b8d6fe573\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:25.891064 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.891038 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnd7k\" (UniqueName: \"kubernetes.io/projected/8b3a80ba-8916-4ce4-8a8b-401b8d6fe573-kube-api-access-bnd7k\") pod \"limitador-operator-controller-manager-859d7b78bd-8btrn\" (UID: \"8b3a80ba-8916-4ce4-8a8b-401b8d6fe573\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:25.900656 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.900630 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnd7k\" (UniqueName: \"kubernetes.io/projected/8b3a80ba-8916-4ce4-8a8b-401b8d6fe573-kube-api-access-bnd7k\") pod \"limitador-operator-controller-manager-859d7b78bd-8btrn\" (UID: \"8b3a80ba-8916-4ce4-8a8b-401b8d6fe573\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:25.985200 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:25.985141 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:26.115848 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:26.115827 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn"] Jul 17 00:23:26.118507 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:23:26.118481 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3a80ba_8916_4ce4_8a8b_401b8d6fe573.slice/crio-8a0c17a63a8edd3a53c092eb4f7467166ce09e7deff65f3230c90638f341c86e WatchSource:0}: Error finding container 8a0c17a63a8edd3a53c092eb4f7467166ce09e7deff65f3230c90638f341c86e: Status 404 returned error can't find the container with id 8a0c17a63a8edd3a53c092eb4f7467166ce09e7deff65f3230c90638f341c86e Jul 17 00:23:26.120444 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:26.120425 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 00:23:26.854962 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:26.854928 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" event={"ID":"8b3a80ba-8916-4ce4-8a8b-401b8d6fe573","Type":"ContainerStarted","Data":"8a0c17a63a8edd3a53c092eb4f7467166ce09e7deff65f3230c90638f341c86e"} Jul 17 00:23:26.952475 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:26.952448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:27.055590 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:27.055562 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:27.428149 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:27.428118 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:27.562940 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:27.562906 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:28.864897 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:28.864865 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" event={"ID":"8b3a80ba-8916-4ce4-8a8b-401b8d6fe573","Type":"ContainerStarted","Data":"81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c"} Jul 17 00:23:28.865312 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:28.864996 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:28.881899 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:28.881852 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" podStartSLOduration=1.523878165 podStartE2EDuration="3.881840001s" totalImagesPullingTime="2.357961836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:23:28.87943251 +0000 UTC m=+381.625941542" watchObservedRunningTime="2026-07-17 00:23:28.881840001 +0000 UTC m=+381.628349093" Jul 17 00:23:28.915030 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:28.915007 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:29.015724 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:29.015705 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:29.074821 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:29.074790 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:31.181372 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:31.181329 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:31.292223 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:31.292177 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:33.612752 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:33.612725 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:33.952556 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:33.952481 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:36.543807 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:36.543778 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:36.603168 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:36.603132 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:36.947861 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:36.947834 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:37.047625 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:37.047602 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:39.871015 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:39.870987 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:42.751239 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:42.751209 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:42.820847 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:42.820822 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:44.428836 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.428803 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn"] Jul 17 00:23:44.429308 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.429008 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" containerName="manager" containerID="cri-o://81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c" gracePeriod=2 Jul 17 00:23:44.453088 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:23:44.453059 2572 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"8b3a80ba-8916-4ce4-8a8b-401b8d6fe573\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-8btrn\": pods \"limitador-operator-controller-manager-859d7b78bd-8btrn\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:44.457181 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.457159 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn"] Jul 17 00:23:44.465743 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.465721 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9"] Jul 17 00:23:44.466079 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.466061 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" containerName="manager" Jul 17 00:23:44.466079 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.466078 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" containerName="manager" Jul 17 00:23:44.466256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.466151 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" containerName="manager" Jul 17 00:23:44.468946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.468929 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" Jul 17 00:23:44.471316 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:23:44.471289 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8btrn\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:44.478368 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.478346 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9"] Jul 17 00:23:44.517099 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.517076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn44z\" (UniqueName: \"kubernetes.io/projected/b9575fa0-65d6-46c1-90ee-67f955c0b423-kube-api-access-gn44z\") pod \"limitador-operator-controller-manager-859d7b78bd-dcst9\" (UID: \"b9575fa0-65d6-46c1-90ee-67f955c0b423\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" Jul 17 00:23:44.617650 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.617625 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn44z\" (UniqueName: \"kubernetes.io/projected/b9575fa0-65d6-46c1-90ee-67f955c0b423-kube-api-access-gn44z\") pod \"limitador-operator-controller-manager-859d7b78bd-dcst9\" (UID: \"b9575fa0-65d6-46c1-90ee-67f955c0b423\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" Jul 17 00:23:44.627839 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.627814 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn44z\" (UniqueName: \"kubernetes.io/projected/b9575fa0-65d6-46c1-90ee-67f955c0b423-kube-api-access-gn44z\") pod \"limitador-operator-controller-manager-859d7b78bd-dcst9\" (UID: \"b9575fa0-65d6-46c1-90ee-67f955c0b423\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" Jul 17 00:23:44.663421 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.663402 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:44.665643 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:23:44.665622 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8btrn\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:44.718268 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.718222 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnd7k\" (UniqueName: \"kubernetes.io/projected/8b3a80ba-8916-4ce4-8a8b-401b8d6fe573-kube-api-access-bnd7k\") pod \"8b3a80ba-8916-4ce4-8a8b-401b8d6fe573\" (UID: \"8b3a80ba-8916-4ce4-8a8b-401b8d6fe573\") " Jul 17 00:23:44.720196 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.720174 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3a80ba-8916-4ce4-8a8b-401b8d6fe573-kube-api-access-bnd7k" (OuterVolumeSpecName: "kube-api-access-bnd7k") pod "8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" (UID: "8b3a80ba-8916-4ce4-8a8b-401b8d6fe573"). InnerVolumeSpecName "kube-api-access-bnd7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 00:23:44.803677 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.803661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" Jul 17 00:23:44.818682 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.818663 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnd7k\" (UniqueName: \"kubernetes.io/projected/8b3a80ba-8916-4ce4-8a8b-401b8d6fe573-kube-api-access-bnd7k\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 00:23:44.926865 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.926843 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9"] Jul 17 00:23:44.929797 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:23:44.929760 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9575fa0_65d6_46c1_90ee_67f955c0b423.slice/crio-94a6350d2923fb84b6d5eedc11e62db629c9d68072c8b85b3234f7c958da7395 WatchSource:0}: Error finding container 94a6350d2923fb84b6d5eedc11e62db629c9d68072c8b85b3234f7c958da7395: Status 404 returned error can't find the container with id 94a6350d2923fb84b6d5eedc11e62db629c9d68072c8b85b3234f7c958da7395 Jul 17 00:23:44.932851 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.932826 2572 generic.go:356] "Generic (PLEG): container finished" podID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" containerID="81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c" exitCode=0 Jul 17 00:23:44.932974 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.932867 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:44.932974 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.932875 2572 scope.go:117] "RemoveContainer" containerID="81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c" Jul 17 00:23:44.935084 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:23:44.935057 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8btrn\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:44.942954 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.942939 2572 scope.go:117] "RemoveContainer" containerID="81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c" Jul 17 00:23:44.943229 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:23:44.943201 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c\": container with ID starting with 81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c not found: ID does not exist" containerID="81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c" Jul 17 00:23:44.943327 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:44.943236 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c"} err="failed to get container status \"81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c\": rpc error: code = NotFound desc = could not find container \"81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c\": container with ID starting with 81474b4eae31e2c1481a3695e9e37ade374ee9d362247e6af42554363450a61c not found: ID does not exist" Jul 17 00:23:44.943444 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:23:44.943422 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8btrn\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:45.938808 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:45.938773 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" event={"ID":"b9575fa0-65d6-46c1-90ee-67f955c0b423","Type":"ContainerStarted","Data":"33996f6afd3e8528361eb33ff71f1ff96a8761d45486e12642d6ac733d8e0e32"} Jul 17 00:23:45.939170 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:45.938844 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" event={"ID":"b9575fa0-65d6-46c1-90ee-67f955c0b423","Type":"ContainerStarted","Data":"94a6350d2923fb84b6d5eedc11e62db629c9d68072c8b85b3234f7c958da7395"} Jul 17 00:23:45.939170 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:45.938921 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" Jul 17 00:23:45.969724 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:45.969518 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" podStartSLOduration=1.96950438 podStartE2EDuration="1.96950438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:23:45.967995442 +0000 UTC m=+398.714504474" watchObservedRunningTime="2026-07-17 00:23:45.96950438 +0000 UTC m=+398.716013414" Jul 17 00:23:45.970058 ip-10-0-135-32 kubenswrapper[2572]: E0717 00:23:45.970033 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8btrn\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8btrn" Jul 17 00:23:46.087308 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:46.087286 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b3a80ba-8916-4ce4-8a8b-401b8d6fe573" path="/var/lib/kubelet/pods/8b3a80ba-8916-4ce4-8a8b-401b8d6fe573/volumes" Jul 17 00:23:46.570678 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:46.570650 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:46.673696 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:46.673652 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:47.161960 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:47.161925 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:47.266480 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:47.266445 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:49.564355 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:49.564320 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:49.623380 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:49.623350 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:54.818299 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:54.818270 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:54.905022 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:54.904993 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:56.739168 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:56.739129 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:23:56.945084 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:23:56.945060 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dcst9" Jul 17 00:24:03.074303 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:03.074255 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:03.178723 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:03.178695 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:24.331393 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:24.331357 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:24.633051 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:24.633024 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:39.891173 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:39.891098 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:39.996578 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:39.996547 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:40.050230 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.050197 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:40.201003 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.200930 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:24:40.203152 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.203132 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:40.205323 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.205303 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 00:24:40.205424 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.205329 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8kgg7\"" Jul 17 00:24:40.215848 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.215815 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:24:40.241780 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.241754 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:24:40.344627 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.344585 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/70c2366c-7920-4156-8b71-e9484a72a488-config-file\") pod \"limitador-limitador-54d6d988d5-qhtjf\" (UID: \"70c2366c-7920-4156-8b71-e9484a72a488\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:40.344792 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.344654 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jvb\" (UniqueName: \"kubernetes.io/projected/70c2366c-7920-4156-8b71-e9484a72a488-kube-api-access-28jvb\") pod \"limitador-limitador-54d6d988d5-qhtjf\" (UID: \"70c2366c-7920-4156-8b71-e9484a72a488\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:40.446042 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.446000 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/70c2366c-7920-4156-8b71-e9484a72a488-config-file\") pod \"limitador-limitador-54d6d988d5-qhtjf\" (UID: \"70c2366c-7920-4156-8b71-e9484a72a488\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:40.446206 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.446170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jvb\" (UniqueName: \"kubernetes.io/projected/70c2366c-7920-4156-8b71-e9484a72a488-kube-api-access-28jvb\") pod \"limitador-limitador-54d6d988d5-qhtjf\" (UID: \"70c2366c-7920-4156-8b71-e9484a72a488\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:40.446664 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.446646 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/70c2366c-7920-4156-8b71-e9484a72a488-config-file\") pod \"limitador-limitador-54d6d988d5-qhtjf\" (UID: \"70c2366c-7920-4156-8b71-e9484a72a488\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:40.454602 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.454538 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jvb\" (UniqueName: \"kubernetes.io/projected/70c2366c-7920-4156-8b71-e9484a72a488-kube-api-access-28jvb\") pod \"limitador-limitador-54d6d988d5-qhtjf\" (UID: \"70c2366c-7920-4156-8b71-e9484a72a488\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:40.519855 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.519821 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:40.651954 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.651935 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:24:40.654437 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:24:40.654410 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c2366c_7920_4156_8b71_e9484a72a488.slice/crio-b69b58d389762171b861d8b5787c48e340f63dd511dc1975ac7c128bb577623e WatchSource:0}: Error finding container b69b58d389762171b861d8b5787c48e340f63dd511dc1975ac7c128bb577623e: Status 404 returned error can't find the container with id b69b58d389762171b861d8b5787c48e340f63dd511dc1975ac7c128bb577623e Jul 17 00:24:40.719158 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.719071 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:40.797222 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:40.797190 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:41.174423 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:41.174392 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" event={"ID":"70c2366c-7920-4156-8b71-e9484a72a488","Type":"ContainerStarted","Data":"b69b58d389762171b861d8b5787c48e340f63dd511dc1975ac7c128bb577623e"} Jul 17 00:24:43.185539 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:43.185505 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" event={"ID":"70c2366c-7920-4156-8b71-e9484a72a488","Type":"ContainerStarted","Data":"e4f15bf1f89b76110dd6bc7fdad0df317d45f5ce00ae8b85aac4a514b73ebe2b"} Jul 17 00:24:43.185872 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:43.185622 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:43.203040 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:43.202996 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" podStartSLOduration=0.797204208 podStartE2EDuration="3.202979626s" totalImagesPullingTime="2.405775418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:24:43.200739696 +0000 UTC m=+455.947248741" watchObservedRunningTime="2026-07-17 00:24:43.202979626 +0000 UTC m=+455.949488664" Jul 17 00:24:54.191593 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:54.191568 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-qhtjf" Jul 17 00:24:57.597872 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:57.597834 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:57.696836 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:57.696809 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:24:58.026979 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:24:58.026904 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:10.708959 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:10.708923 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:11.042043 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:11.041965 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:16.992178 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:16.992146 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:17.109876 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:17.109847 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:17.184971 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:17.184939 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:20.344770 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:20.344728 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:20.444784 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:20.444759 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:45.083721 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:45.083690 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:52.169045 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:52.169013 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:52.292170 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:52.292139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:52.837474 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:52.837445 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:53.036545 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:53.036510 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:53.404239 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:53.404207 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:53.504504 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:53.504474 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:25:53.664307 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:25:53.664230 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:26:04.618998 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:04.618924 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:26:06.370299 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:06.370263 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:26:06.472932 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:06.472902 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:26:06.581271 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:06.581240 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:26:22.832995 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:22.832959 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:26:23.303955 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:23.303922 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:26:35.224248 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:35.224212 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:26:35.906210 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:35.906180 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:26:39.526351 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:39.526317 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:26:40.486648 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:40.486614 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:26:48.689777 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.689745 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm"] Jul 17 00:26:48.692103 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.692087 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.695357 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.695334 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 17 00:26:48.695483 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.695355 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nnsb6\"" Jul 17 00:26:48.695483 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.695394 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 00:26:48.695483 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.695356 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 00:26:48.702398 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.702379 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm"] Jul 17 00:26:48.717107 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.717082 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:26:48.832280 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.832259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.832384 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.832295 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.832384 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.832319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6nc7\" (UniqueName: \"kubernetes.io/projected/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-kube-api-access-p6nc7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.832460 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.832415 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.832460 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.832433 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.832522 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.832478 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.933094 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.933059 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.933205 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.933102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6nc7\" (UniqueName: \"kubernetes.io/projected/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-kube-api-access-p6nc7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.933205 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.933152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.933205 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.933179 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.933364 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.933233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.933364 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.933301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.933522 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.933440 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.933522 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.933502 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.933629 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.933541 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.935376 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.935360 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.935637 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.935623 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:48.939963 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:48.939915 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6nc7\" (UniqueName: \"kubernetes.io/projected/b58a8f79-4a7b-42eb-bb97-fc3be9255b39-kube-api-access-p6nc7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm\" (UID: \"b58a8f79-4a7b-42eb-bb97-fc3be9255b39\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:49.003709 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:49.003686 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:49.159112 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:49.159089 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm"] Jul 17 00:26:49.162326 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:26:49.162302 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58a8f79_4a7b_42eb_bb97_fc3be9255b39.slice/crio-19eb4ea8bca2dd7c5b8aacb25f0963a1a17497bbf13d6bcc923fa3ce4b34502e WatchSource:0}: Error finding container 19eb4ea8bca2dd7c5b8aacb25f0963a1a17497bbf13d6bcc923fa3ce4b34502e: Status 404 returned error can't find the container with id 19eb4ea8bca2dd7c5b8aacb25f0963a1a17497bbf13d6bcc923fa3ce4b34502e Jul 17 00:26:49.738389 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:49.738352 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" event={"ID":"b58a8f79-4a7b-42eb-bb97-fc3be9255b39","Type":"ContainerStarted","Data":"19eb4ea8bca2dd7c5b8aacb25f0963a1a17497bbf13d6bcc923fa3ce4b34502e"} Jul 17 00:26:50.194748 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:50.194721 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:26:50.745076 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:50.745039 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" event={"ID":"b58a8f79-4a7b-42eb-bb97-fc3be9255b39","Type":"ContainerStarted","Data":"2682d8b9329c269e35c9ea75a9c34d02177d8a87986dacb2f82c3ba18d65f380"} Jul 17 00:26:50.765090 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:50.765048 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" podStartSLOduration=1.7967826869999999 podStartE2EDuration="2.765034187s" totalImagesPullingTime="968.2515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:26:50.762567055 +0000 UTC m=+583.509076085" watchObservedRunningTime="2026-07-17 00:26:50.765034187 +0000 UTC m=+583.511543220" Jul 17 00:26:53.412261 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:53.412231 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:26:55.792902 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:55.792852 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:26:59.003866 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:59.003822 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:59.004292 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:59.003915 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:59.016111 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:59.016091 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:26:59.797906 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:26:59.797853 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm" Jul 17 00:27:05.393547 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.393511 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp"] Jul 17 00:27:05.396254 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.396237 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.398648 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.398630 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 17 00:27:05.406917 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.406879 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp"] Jul 17 00:27:05.416617 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.416590 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:27:05.560037 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.560011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.560156 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.560046 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b8ad88-1922-4533-bc52-9574420d058d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.560156 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.560088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.560235 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.560170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.560235 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.560199 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.560306 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.560241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6mq\" (UniqueName: \"kubernetes.io/projected/93b8ad88-1922-4533-bc52-9574420d058d-kube-api-access-xm6mq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.660821 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.660759 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6mq\" (UniqueName: \"kubernetes.io/projected/93b8ad88-1922-4533-bc52-9574420d058d-kube-api-access-xm6mq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.660821 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.660806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.661007 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.660826 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b8ad88-1922-4533-bc52-9574420d058d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.661007 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.660856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.661007 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.660876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.661007 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.660915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.661249 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.661229 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.661303 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.661264 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.661347 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.661299 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.663227 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.663203 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b8ad88-1922-4533-bc52-9574420d058d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.663349 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.663331 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b8ad88-1922-4533-bc52-9574420d058d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.669584 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.669561 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6mq\" (UniqueName: \"kubernetes.io/projected/93b8ad88-1922-4533-bc52-9574420d058d-kube-api-access-xm6mq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp\" (UID: \"93b8ad88-1922-4533-bc52-9574420d058d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.705378 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.705358 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:05.831776 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:05.831712 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp"] Jul 17 00:27:05.833827 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:27:05.833801 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93b8ad88_1922_4533_bc52_9574420d058d.slice/crio-d90aa492b6e86d58994367f21194fc0904a3e619c4afacdd1c880702b9134ec3 WatchSource:0}: Error finding container d90aa492b6e86d58994367f21194fc0904a3e619c4afacdd1c880702b9134ec3: Status 404 returned error can't find the container with id d90aa492b6e86d58994367f21194fc0904a3e619c4afacdd1c880702b9134ec3 Jul 17 00:27:06.239147 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:06.239113 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:27:06.340368 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:06.340298 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:27:06.821098 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:06.821065 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" event={"ID":"93b8ad88-1922-4533-bc52-9574420d058d","Type":"ContainerStarted","Data":"e6199fb6a1b2aa7f1c112fbfc7bad722622f5c7430ff32ced1469acce24e9de8"} Jul 17 00:27:06.821542 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:06.821113 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" event={"ID":"93b8ad88-1922-4533-bc52-9574420d058d","Type":"ContainerStarted","Data":"d90aa492b6e86d58994367f21194fc0904a3e619c4afacdd1c880702b9134ec3"} Jul 17 00:27:06.840347 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:06.840304 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" podStartSLOduration=1.6734993139999998 podStartE2EDuration="1.840291386s" totalImagesPullingTime="166.792072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:27:06.837983135 +0000 UTC m=+599.584492178" watchObservedRunningTime="2026-07-17 00:27:06.840291386 +0000 UTC m=+599.586800419" Jul 17 00:27:07.799816 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:07.799786 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:27:15.705630 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:15.705597 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:15.706064 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:15.705867 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:15.717993 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:15.717971 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:15.869080 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:15.869052 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp" Jul 17 00:27:25.637114 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:25.637059 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:27:25.737414 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:25.737386 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:27:54.606310 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:54.606278 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:27:54.687005 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:54.686980 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:27:55.809286 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:55.809252 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:27:55.938800 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:27:55.938769 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:28:17.894583 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:28:17.894551 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:28:26.794041 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:28:26.794010 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:28:36.393414 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:28:36.393381 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:28:45.500604 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:28:45.500569 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:29:03.362211 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:29:03.362117 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:29:38.089681 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:29:38.089650 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:29:38.143167 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:29:38.143130 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:01.306981 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:01.306946 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:02.632833 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:02.632801 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:27.693648 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:27.693613 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:30:30.670027 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:30.669951 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:30.770695 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:30.770663 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:40.299222 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:40.299192 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:30:45.032928 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:45.032877 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:45.132516 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:45.132483 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:45.439778 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:45.439722 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:50.903093 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:50.903059 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:51.003539 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:51.003507 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:51.103432 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:51.103402 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:51.226023 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:51.225951 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:57.125338 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:57.125201 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:30:57.215631 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:30:57.215601 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:31:12.898067 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:31:12.898028 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:31:24.293253 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:31:24.293214 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:31:34.492157 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:31:34.492127 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:31:42.195903 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:31:42.195870 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:31:52.388812 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:31:52.388778 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:32:03.092515 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:32:03.092435 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:32:14.388804 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:32:14.388769 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:32:24.697122 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:32:24.697086 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:32:36.618478 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:32:36.618445 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:32:41.409170 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:32:41.409137 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:32:46.093758 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:32:46.093723 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:32:56.990464 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:32:56.990430 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:33:21.386554 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:33:21.386516 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:33:31.515401 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:33:31.515319 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:33:56.094851 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:33:56.094824 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:34:05.398933 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:34:05.398883 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:34:16.199802 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:34:16.199775 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:34:26.595084 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:34:26.595052 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:36:07.984953 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:36:07.984913 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:36:14.688954 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:36:14.688920 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:36:27.500870 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:36:27.500827 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:36:35.440122 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:36:35.440047 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:36:47.992121 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:36:47.992085 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:36:57.191905 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:36:57.191851 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:37:07.598705 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:37:07.598673 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:37:17.394272 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:37:17.394236 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:37:27.599932 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:37:27.599897 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:37:39.097494 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:37:39.097464 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:37:48.513904 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:37:48.513853 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:37:58.592032 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:37:58.591947 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:38:20.598698 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:38:20.598668 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:38:29.997700 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:38:29.997666 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:38:38.898547 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:38:38.898509 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:38:40.641946 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:38:40.641915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:38:42.412145 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:38:42.412109 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:38:50.887346 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:38:50.887314 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:39:07.711045 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:39:07.711003 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:39:16.890847 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:39:16.890812 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:39:26.192924 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:39:26.192877 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:39:34.494809 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:39:34.494735 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:39:44.996531 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:39:44.996500 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:39:53.794080 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:39:53.794045 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:40:07.188512 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:40:07.188470 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:40:16.898680 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:40:16.898642 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:40:25.898533 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:40:25.898505 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:40:38.492570 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:40:38.492534 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:40:44.253159 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:40:44.253127 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:40:48.596336 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:40:48.596305 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:41:00.702668 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:00.702594 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:41:12.599023 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:12.598993 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:41:24.502705 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:24.502662 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:41:32.994443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:32.994405 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:41:36.428590 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:36.428559 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:41:42.901934 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:42.901880 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:41:45.891159 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:45.891122 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:41:51.156975 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:51.156940 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:41:54.595484 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:54.595449 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:41:56.933267 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:56.933234 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:41:59.740262 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:59.740230 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:41:59.840243 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:41:59.840212 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:00.330969 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:00.330938 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:04.891851 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:04.891814 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:42:05.964021 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:05.963989 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:06.067175 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:06.067145 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:06.172872 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:06.172846 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:06.233353 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:06.233290 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:07.265580 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:07.265544 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:07.347385 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:07.347351 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:07.644297 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:07.644259 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:07.949958 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:07.949867 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:08.137170 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:08.137132 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:08.238785 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:08.238706 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:09.901617 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:09.901588 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:42:15.594489 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:42:15.594454 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:43:20.978709 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:43:20.978674 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:12.476427 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:12.476381 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:14.527447 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:14.527410 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:15.092092 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:15.092057 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:45:18.026459 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:18.026378 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:18.460788 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:18.460753 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:23.821775 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:23.821741 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:23.883377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:23.883345 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:24.070230 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:24.070186 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:24.320944 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:24.320906 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:24.421292 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:24.421258 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:24.783517 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:24.783486 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:24.883396 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:24.883363 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:25.587971 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:25.587931 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:25.681767 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:25.681730 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:26.285986 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:26.285943 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:26.386967 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:26.386929 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:28.297807 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:28.297774 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:45:40.782901 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:40.782826 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:40.882908 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:40.882859 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:45:44.251416 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:45:44.251386 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:30.179947 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:30.179908 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:30.349117 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:30.349085 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:33.492927 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:33.492871 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:46:35.882814 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:35.882780 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:36.012281 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:36.012243 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:36.654062 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:36.654034 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:41.724179 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:41.724143 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:41.822184 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:41.822151 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:41.950118 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:41.950082 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:42.010132 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:42.010058 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:43.051437 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:43.051399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:43.130940 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:43.130866 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:43.203795 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:43.203731 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:43.602575 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:43.602546 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:43.725103 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:43.725074 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:43.820093 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:43.820067 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:45.368522 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:45.368488 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:45.993260 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:45.993229 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:46:57.425882 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:57.425829 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:46:57.476574 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:46:57.476545 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:48:33.506745 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:48:33.506666 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:49:32.580471 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:49:32.580436 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:50:25.616851 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:50:25.616815 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:50:28.278249 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:50:28.278217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:50:28.643773 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:50:28.643746 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:50:28.743246 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:50:28.743215 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:52:48.568773 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:52:48.568743 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:52:48.655965 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:52:48.655936 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:52:51.294400 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:52:51.294368 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:52:54.344023 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:52:54.343989 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:52:54.604238 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:52:54.604158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:00.028685 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:00.028591 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:00.224024 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:00.223994 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:00.311083 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:00.311014 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:01.409954 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:01.409921 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:01.497955 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:01.497916 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:01.620282 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:01.620245 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:02.334081 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:02.334048 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:02.434245 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:02.434219 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:03.987768 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:03.987735 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:53:10.245943 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:10.245916 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:10.389671 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:10.389628 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:53:13.872443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:53:13.872410 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:55:26.451715 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:55:26.451681 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:55:26.543446 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:55:26.543413 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:55:36.470023 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:55:36.469991 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:55:38.051078 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:55:38.051047 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:05.985054 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:05.984964 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:06.081358 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:06.081317 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:08.698394 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:08.698359 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:56:11.784070 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:11.784034 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:11.885283 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:11.885248 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:17.430803 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:17.430763 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:17.531530 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:17.531491 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:17.895152 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:17.895119 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:17.952660 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:17.952618 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:18.024984 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:18.024956 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:19.479906 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:19.479853 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:25.006072 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:25.006039 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:25.060830 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:25.060798 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:25.161388 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:25.161344 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:25.263277 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:25.263202 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:25.361474 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:25.361440 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:25.756755 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:25.756721 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:28.105638 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:28.105608 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:51.609723 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:51.609684 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:53.614201 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:53.614170 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:57.407772 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:57.407738 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:57.461450 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:57.461410 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:57.538124 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:57.538091 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:56:57.662505 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:56:57.662407 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:13.632083 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:13.632037 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:19.110812 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:19.110771 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:19.213569 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:19.213533 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:19.309032 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:19.308997 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:29.816648 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:29.816551 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:30.610305 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:30.610276 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:36.377837 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:36.377793 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:36.481141 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:36.481112 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:36.577979 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:36.577951 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:36.677780 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:36.677706 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:37.376822 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:37.376785 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:37.477278 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:37.477247 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:37.575987 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:37.575949 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:39.076747 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:39.076710 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:39.696950 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:39.696909 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:57:44.744208 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:44.744173 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:44.844352 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:44.844318 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:45.372075 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:45.372043 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:48.688738 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:48.688703 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:57:51.311012 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:51.310970 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:51.407222 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:51.407190 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:51.533681 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:51.533647 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:51.827248 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:51.827204 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:52.529190 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:52.529162 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:54.899293 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:54.899261 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:54.997866 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:54.997833 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:55.059243 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:55.059202 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:55.159610 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:55.159521 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:55.572249 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:55.572169 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:55.673046 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:55.673012 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:59.360586 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:59.360543 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:57:59.791212 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:59.791133 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:57:59.969468 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:57:59.969437 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:00.064256 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:00.064180 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:00.412828 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:00.412790 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:06.102192 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:06.102149 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:06.181191 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:06.181156 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:17.074333 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:17.074293 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:19.986733 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:19.986697 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:58:21.533476 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:21.533445 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:27.201202 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:27.201162 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:27.254577 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:27.254552 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:32.190237 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:32.190203 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:32.412497 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:32.412467 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:32.532817 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:32.532706 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:32.632724 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:32.632686 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:32.744716 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:32.744685 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:38.288964 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:38.288929 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:38.389137 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:38.389110 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:38.489462 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:38.489422 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:38.590109 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:38.590018 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:38.704476 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:38.704443 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:44.462471 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:44.462439 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:44.518089 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:44.518053 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:51.154504 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:51.154468 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:51.440157 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:51.440087 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:51.927118 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:51.927076 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:54.153811 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:54.153775 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:54.236014 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:54.235965 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:54.296940 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:54.296907 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:54.453274 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:54.453190 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:54.904627 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:54.904590 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:55.005223 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:55.005194 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:58:58.638613 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:58:58.638531 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:59:01.190065 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.190029 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:59:01.890761 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.890728 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc"] Jul 17 00:59:01.894883 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.894864 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:01.897381 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.897357 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0a075aab\"/\"e2e-lifecycle-model-0a075aab-kserve-self-signed-certs\"" Jul 17 00:59:01.898472 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.898447 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0a075aab\"/\"openshift-service-ca.crt\"" Jul 17 00:59:01.898565 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.898449 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0a075aab\"/\"default-dockercfg-6tsqp\"" Jul 17 00:59:01.898565 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.898522 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0a075aab\"/\"kube-root-ca.crt\"" Jul 17 00:59:01.905152 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.905133 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc"] Jul 17 00:59:01.915766 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.915740 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:59:01.954320 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.954297 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-model-cache\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:01.954443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.954341 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-dshm\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:01.954443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.954361 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-tmp-dir\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:01.954443 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.954381 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57150ed4-c383-48d6-9673-bafa28d5d177-tls-certs\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:01.954601 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.954472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc6b5\" (UniqueName: \"kubernetes.io/projected/57150ed4-c383-48d6-9673-bafa28d5d177-kube-api-access-wc6b5\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:01.954601 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:01.954501 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-home\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.055003 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.054976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-dshm\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.055107 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.055008 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-tmp-dir\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.055107 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.055025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57150ed4-c383-48d6-9673-bafa28d5d177-tls-certs\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.055107 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.055078 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6b5\" (UniqueName: \"kubernetes.io/projected/57150ed4-c383-48d6-9673-bafa28d5d177-kube-api-access-wc6b5\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.055107 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.055098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-home\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.055262 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.055130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-model-cache\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.055428 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.055406 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-tmp-dir\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.055523 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.055504 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-model-cache\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.055566 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.055535 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-home\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.057254 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.057239 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-dshm\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.057598 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.057581 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57150ed4-c383-48d6-9673-bafa28d5d177-tls-certs\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.062705 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.062686 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6b5\" (UniqueName: \"kubernetes.io/projected/57150ed4-c383-48d6-9673-bafa28d5d177-kube-api-access-wc6b5\") pod \"e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.205794 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.205732 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:02.332795 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.332765 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc"] Jul 17 00:59:02.336018 ip-10-0-135-32 kubenswrapper[2572]: W0717 00:59:02.335994 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57150ed4_c383_48d6_9673_bafa28d5d177.slice/crio-e2406103e90154631f1a4d1e67a8c4254541bc46d07d3fefa778b880c0a88e26 WatchSource:0}: Error finding container e2406103e90154631f1a4d1e67a8c4254541bc46d07d3fefa778b880c0a88e26: Status 404 returned error can't find the container with id e2406103e90154631f1a4d1e67a8c4254541bc46d07d3fefa778b880c0a88e26 Jul 17 00:59:02.383185 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:02.383162 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" event={"ID":"57150ed4-c383-48d6-9673-bafa28d5d177","Type":"ContainerStarted","Data":"e2406103e90154631f1a4d1e67a8c4254541bc46d07d3fefa778b880c0a88e26"} Jul 17 00:59:03.388164 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:03.388131 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" event={"ID":"57150ed4-c383-48d6-9673-bafa28d5d177","Type":"ContainerStarted","Data":"73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b"} Jul 17 00:59:03.404714 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:03.404665 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" podStartSLOduration=2.40465105 podStartE2EDuration="2.40465105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 00:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 00:59:03.402486638 +0000 UTC m=+2516.148995674" watchObservedRunningTime="2026-07-17 00:59:03.40465105 +0000 UTC m=+2516.151160083" Jul 17 00:59:10.088386 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:10.088354 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:59:12.206383 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:12.206337 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:12.206383 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:12.206391 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:12.218397 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:12.218374 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:12.441620 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:12.441592 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 00:59:28.399151 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:28.399119 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:59:41.340783 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:41.340746 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:59:42.135377 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:42.135341 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:59:45.496333 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:45.496303 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 00:59:46.740143 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:46.740107 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:59:49.594168 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:49.594134 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 00:59:51.773267 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:51.773233 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc"] Jul 17 00:59:52.045022 ip-10-0-135-32 kubenswrapper[2572]: I0717 00:59:52.044933 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:06.793440 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:06.793404 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" podUID="57150ed4-c383-48d6-9673-bafa28d5d177" containerName="main" containerID="cri-o://73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b" gracePeriod=45 Jul 17 01:00:07.045692 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.045636 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 01:00:07.117466 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.117439 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-dshm\") pod \"57150ed4-c383-48d6-9673-bafa28d5d177\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " Jul 17 01:00:07.117603 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.117473 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc6b5\" (UniqueName: \"kubernetes.io/projected/57150ed4-c383-48d6-9673-bafa28d5d177-kube-api-access-wc6b5\") pod \"57150ed4-c383-48d6-9673-bafa28d5d177\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " Jul 17 01:00:07.117603 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.117498 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57150ed4-c383-48d6-9673-bafa28d5d177-tls-certs\") pod \"57150ed4-c383-48d6-9673-bafa28d5d177\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " Jul 17 01:00:07.117603 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.117548 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-tmp-dir\") pod \"57150ed4-c383-48d6-9673-bafa28d5d177\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " Jul 17 01:00:07.117603 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.117597 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-home\") pod \"57150ed4-c383-48d6-9673-bafa28d5d177\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " Jul 17 01:00:07.117801 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.117673 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-model-cache\") pod \"57150ed4-c383-48d6-9673-bafa28d5d177\" (UID: \"57150ed4-c383-48d6-9673-bafa28d5d177\") " Jul 17 01:00:07.117862 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.117821 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57150ed4-c383-48d6-9673-bafa28d5d177" (UID: "57150ed4-c383-48d6-9673-bafa28d5d177"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:00:07.117972 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.117951 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-home" (OuterVolumeSpecName: "home") pod "57150ed4-c383-48d6-9673-bafa28d5d177" (UID: "57150ed4-c383-48d6-9673-bafa28d5d177"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:00:07.118030 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.117969 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-model-cache" (OuterVolumeSpecName: "model-cache") pod "57150ed4-c383-48d6-9673-bafa28d5d177" (UID: "57150ed4-c383-48d6-9673-bafa28d5d177"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:00:07.118066 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.118035 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:00:07.119684 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.119666 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57150ed4-c383-48d6-9673-bafa28d5d177-kube-api-access-wc6b5" (OuterVolumeSpecName: "kube-api-access-wc6b5") pod "57150ed4-c383-48d6-9673-bafa28d5d177" (UID: "57150ed4-c383-48d6-9673-bafa28d5d177"). InnerVolumeSpecName "kube-api-access-wc6b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:00:07.119743 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.119686 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-dshm" (OuterVolumeSpecName: "dshm") pod "57150ed4-c383-48d6-9673-bafa28d5d177" (UID: "57150ed4-c383-48d6-9673-bafa28d5d177"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:00:07.119777 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.119764 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57150ed4-c383-48d6-9673-bafa28d5d177-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57150ed4-c383-48d6-9673-bafa28d5d177" (UID: "57150ed4-c383-48d6-9673-bafa28d5d177"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:00:07.219136 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.219102 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:00:07.219136 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.219139 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc6b5\" (UniqueName: \"kubernetes.io/projected/57150ed4-c383-48d6-9673-bafa28d5d177-kube-api-access-wc6b5\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:00:07.219323 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.219153 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57150ed4-c383-48d6-9673-bafa28d5d177-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:00:07.219323 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.219163 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:00:07.219323 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.219171 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57150ed4-c383-48d6-9673-bafa28d5d177-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:00:07.681497 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.681466 2572 generic.go:356] "Generic (PLEG): container finished" podID="57150ed4-c383-48d6-9673-bafa28d5d177" containerID="73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b" exitCode=0 Jul 17 01:00:07.681659 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.681512 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" Jul 17 01:00:07.681659 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.681517 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" event={"ID":"57150ed4-c383-48d6-9673-bafa28d5d177","Type":"ContainerDied","Data":"73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b"} Jul 17 01:00:07.681659 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.681546 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc" event={"ID":"57150ed4-c383-48d6-9673-bafa28d5d177","Type":"ContainerDied","Data":"e2406103e90154631f1a4d1e67a8c4254541bc46d07d3fefa778b880c0a88e26"} Jul 17 01:00:07.681659 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.681562 2572 scope.go:117] "RemoveContainer" containerID="73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b" Jul 17 01:00:07.690305 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.690288 2572 scope.go:117] "RemoveContainer" containerID="73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b" Jul 17 01:00:07.690552 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:00:07.690536 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b\": container with ID starting with 73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b not found: ID does not exist" containerID="73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b" Jul 17 01:00:07.690595 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.690560 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b"} err="failed to get container status \"73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b\": rpc error: code = NotFound desc = could not find container \"73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b\": container with ID starting with 73818980d87f3cfa7b80c2879f6a0a4031dca6869833ba05899f28ccef16c83b not found: ID does not exist" Jul 17 01:00:07.700819 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.700796 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc"] Jul 17 01:00:07.704641 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:07.704620 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0a075aab/e2e-lifecycle-model-0a075aab-kserve-846cc56ccf-6zbkc"] Jul 17 01:00:08.087741 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:08.087710 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57150ed4-c383-48d6-9673-bafa28d5d177" path="/var/lib/kubelet/pods/57150ed4-c383-48d6-9673-bafa28d5d177/volumes" Jul 17 01:00:27.327948 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:27.327914 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:27.427959 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:27.427932 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:28.995107 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:28.995002 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:29.047300 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:29.047264 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:34.268487 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:34.268454 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:34.366436 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:34.366388 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:38.931166 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:38.931135 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:43.846305 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:43.846269 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:48.196752 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:48.196692 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:48.297951 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:48.297915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:48.842514 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:48.842487 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:48.901108 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:48.901075 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:00:54.525781 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:54.525748 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:54.585963 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:54.585928 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:54.670001 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:54.669962 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:54.774622 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:54.774593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:00:56.790613 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:00:56.790578 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:01:05.192348 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:05.192312 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:05.523359 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:05.523272 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:26.470037 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:26.470000 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:32.081156 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:32.081116 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:32.181835 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:32.181797 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:32.406117 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:32.406092 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:32.729516 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:32.729445 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:33.101331 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:33.101302 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:35.950851 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:35.950818 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:36.049787 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:36.049744 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:36.231574 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:36.231502 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:36.613507 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:36.613475 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:36.821714 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:36.821685 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:36.921782 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:36.921688 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:40.435473 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:40.435439 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:40.751930 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:40.751841 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:42.410144 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:42.410105 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:42.511578 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:42.511547 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:43.592442 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:43.592407 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:01:43.984467 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:43.984389 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:44.983667 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:44.983626 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:45.085016 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:45.084980 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:48.479903 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:48.479865 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:54.985805 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:54.985772 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:01:55.617359 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.617324 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5"] Jul 17 01:01:55.617684 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.617673 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57150ed4-c383-48d6-9673-bafa28d5d177" containerName="main" Jul 17 01:01:55.617862 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.617686 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="57150ed4-c383-48d6-9673-bafa28d5d177" containerName="main" Jul 17 01:01:55.617862 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.617745 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="57150ed4-c383-48d6-9673-bafa28d5d177" containerName="main" Jul 17 01:01:55.620737 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.620719 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.624178 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.624152 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-27f370\"/\"default-dockercfg-6wcf9\"" Jul 17 01:01:55.624288 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.624201 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-27f370\"/\"test-model-27f370-kserve-self-signed-certs\"" Jul 17 01:01:55.624288 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.624223 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-27f370\"/\"openshift-service-ca.crt\"" Jul 17 01:01:55.624288 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.624201 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-27f370\"/\"kube-root-ca.crt\"" Jul 17 01:01:55.630271 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.630252 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5"] Jul 17 01:01:55.720145 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.720111 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:55.740101 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.740077 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-dshm\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.740228 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.740106 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9t49\" (UniqueName: \"kubernetes.io/projected/1b606a08-fe7b-407b-8d0f-195184e25ec9-kube-api-access-z9t49\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.740228 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.740127 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-model-cache\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.740349 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.740256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-home\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.740349 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.740294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b606a08-fe7b-407b-8d0f-195184e25ec9-tls-certs\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.740349 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.740332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-tmp-dir\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.841612 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.841583 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-dshm\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.841612 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.841613 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9t49\" (UniqueName: \"kubernetes.io/projected/1b606a08-fe7b-407b-8d0f-195184e25ec9-kube-api-access-z9t49\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.841769 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.841631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-model-cache\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.841769 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.841687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-home\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.841769 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.841705 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b606a08-fe7b-407b-8d0f-195184e25ec9-tls-certs\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.841769 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.841727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-tmp-dir\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.842109 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.842076 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-model-cache\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.842231 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.842121 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-tmp-dir\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.842231 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.842139 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-home\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.843651 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.843630 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-dshm\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.843962 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.843947 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b606a08-fe7b-407b-8d0f-195184e25ec9-tls-certs\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.849859 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.849841 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9t49\" (UniqueName: \"kubernetes.io/projected/1b606a08-fe7b-407b-8d0f-195184e25ec9-kube-api-access-z9t49\") pod \"test-model-27f370-kserve-cd66ccdc5-wmgf5\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:55.930027 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:55.929964 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:01:56.054426 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:56.054395 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5"] Jul 17 01:01:56.057697 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:01:56.057671 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b606a08_fe7b_407b_8d0f_195184e25ec9.slice/crio-bb4b2353226466c34e73362def178848bcf5d67471f34ac203a89823ff9f49e1 WatchSource:0}: Error finding container bb4b2353226466c34e73362def178848bcf5d67471f34ac203a89823ff9f49e1: Status 404 returned error can't find the container with id bb4b2353226466c34e73362def178848bcf5d67471f34ac203a89823ff9f49e1 Jul 17 01:01:56.174841 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:56.174806 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" event={"ID":"1b606a08-fe7b-407b-8d0f-195184e25ec9","Type":"ContainerStarted","Data":"44d69166e8c215b103d360210ea2a7fa6cfd9499967678bef009b495e809f03b"} Jul 17 01:01:56.174997 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:56.174853 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" event={"ID":"1b606a08-fe7b-407b-8d0f-195184e25ec9","Type":"ContainerStarted","Data":"bb4b2353226466c34e73362def178848bcf5d67471f34ac203a89823ff9f49e1"} Jul 17 01:01:56.192457 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:56.192378 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" podStartSLOduration=1.192363633 podStartE2EDuration="1.192363633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:01:56.191580748 +0000 UTC m=+2688.938089794" watchObservedRunningTime="2026-07-17 01:01:56.192363633 +0000 UTC m=+2688.938872666" Jul 17 01:01:57.571141 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:57.571109 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:01:57.670802 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:01:57.670757 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:02:04.297051 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:04.296975 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:02:05.587799 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:05.587764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:02:05.685487 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:05.685450 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:02:05.930159 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:05.930127 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:02:05.930326 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:05.930178 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:02:05.942144 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:05.942124 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:02:06.231423 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:06.231347 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:02:46.464203 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:46.464166 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:02:52.304797 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.304763 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc"] Jul 17 01:02:52.308206 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.308190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.310849 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.310823 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e1e207\"/\"test-model-e1e207-kserve-self-signed-certs\"" Jul 17 01:02:52.310985 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.310852 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e1e207\"/\"openshift-service-ca.crt\"" Jul 17 01:02:52.311756 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.311735 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e1e207\"/\"kube-root-ca.crt\"" Jul 17 01:02:52.312483 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.312463 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e1e207\"/\"default-dockercfg-ggh9g\"" Jul 17 01:02:52.316850 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.316826 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc"] Jul 17 01:02:52.328112 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.328085 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:02:52.439004 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.438972 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tls-certs\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.439004 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.439005 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw5vx\" (UniqueName: \"kubernetes.io/projected/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-kube-api-access-lw5vx\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.439289 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.439087 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-dshm\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.439289 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.439112 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-model-cache\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.439289 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.439138 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-home\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.439289 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.439225 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tmp-dir\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.540594 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.540559 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tmp-dir\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.540791 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.540614 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tls-certs\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.540791 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.540648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw5vx\" (UniqueName: \"kubernetes.io/projected/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-kube-api-access-lw5vx\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.540791 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.540702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-dshm\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.540791 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.540729 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-model-cache\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.541050 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.540926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-home\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.541050 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.541039 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tmp-dir\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.541146 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.541056 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-model-cache\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.541262 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.541244 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-home\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.543019 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.543000 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-dshm\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.543195 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.543178 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tls-certs\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.547986 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.547963 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw5vx\" (UniqueName: \"kubernetes.io/projected/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-kube-api-access-lw5vx\") pod \"test-model-e1e207-kserve-6784874965-hfmnc\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.618661 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.618631 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:02:52.746818 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:52.746794 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc"] Jul 17 01:02:52.749491 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:02:52.749466 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4047b6b9_a339_4f82_8e12_ec46b1bef1a7.slice/crio-dc8710a207a17033fab98d99e402f0a4649aadacb0944cbec268312333241d99 WatchSource:0}: Error finding container dc8710a207a17033fab98d99e402f0a4649aadacb0944cbec268312333241d99: Status 404 returned error can't find the container with id dc8710a207a17033fab98d99e402f0a4649aadacb0944cbec268312333241d99 Jul 17 01:02:53.438124 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:53.438079 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" event={"ID":"4047b6b9-a339-4f82-8e12-ec46b1bef1a7","Type":"ContainerStarted","Data":"9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82"} Jul 17 01:02:53.438124 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:53.438129 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" event={"ID":"4047b6b9-a339-4f82-8e12-ec46b1bef1a7","Type":"ContainerStarted","Data":"dc8710a207a17033fab98d99e402f0a4649aadacb0944cbec268312333241d99"} Jul 17 01:02:53.454737 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:53.454693 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" podStartSLOduration=1.454679773 podStartE2EDuration="1.454679773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:02:53.45293452 +0000 UTC m=+2746.199443555" watchObservedRunningTime="2026-07-17 01:02:53.454679773 +0000 UTC m=+2746.201188808" Jul 17 01:02:56.992724 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:02:56.992687 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:03:02.619356 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:02.619296 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:03:02.619356 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:02.619360 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:03:02.631657 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:02.631625 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:03:03.493528 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:03.493499 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:03:11.995205 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:11.995165 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:03:31.674813 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:31.674733 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:03:31.724538 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:31.724505 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:03:33.452563 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:33.452533 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5"] Jul 17 01:03:33.476446 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:33.476417 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:03:34.349471 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:34.349438 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc"] Jul 17 01:03:34.365428 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:34.365384 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:03:35.180301 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:35.180270 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:03:35.281947 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:35.281912 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:03:37.493657 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:37.493624 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:03:45.021583 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:45.021536 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:03:48.474763 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.474702 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" podUID="1b606a08-fe7b-407b-8d0f-195184e25ec9" containerName="main" containerID="cri-o://44d69166e8c215b103d360210ea2a7fa6cfd9499967678bef009b495e809f03b" gracePeriod=45 Jul 17 01:03:48.700245 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.700208 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b606a08-fe7b-407b-8d0f-195184e25ec9" containerID="44d69166e8c215b103d360210ea2a7fa6cfd9499967678bef009b495e809f03b" exitCode=0 Jul 17 01:03:48.700409 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.700276 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" event={"ID":"1b606a08-fe7b-407b-8d0f-195184e25ec9","Type":"ContainerDied","Data":"44d69166e8c215b103d360210ea2a7fa6cfd9499967678bef009b495e809f03b"} Jul 17 01:03:48.726901 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.726832 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:03:48.847680 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.847644 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9t49\" (UniqueName: \"kubernetes.io/projected/1b606a08-fe7b-407b-8d0f-195184e25ec9-kube-api-access-z9t49\") pod \"1b606a08-fe7b-407b-8d0f-195184e25ec9\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " Jul 17 01:03:48.847860 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.847695 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-home\") pod \"1b606a08-fe7b-407b-8d0f-195184e25ec9\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " Jul 17 01:03:48.847860 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.847742 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b606a08-fe7b-407b-8d0f-195184e25ec9-tls-certs\") pod \"1b606a08-fe7b-407b-8d0f-195184e25ec9\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " Jul 17 01:03:48.847860 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.847775 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-dshm\") pod \"1b606a08-fe7b-407b-8d0f-195184e25ec9\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " Jul 17 01:03:48.847860 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.847798 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-tmp-dir\") pod \"1b606a08-fe7b-407b-8d0f-195184e25ec9\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " Jul 17 01:03:48.847860 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.847826 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-model-cache\") pod \"1b606a08-fe7b-407b-8d0f-195184e25ec9\" (UID: \"1b606a08-fe7b-407b-8d0f-195184e25ec9\") " Jul 17 01:03:48.848160 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.848076 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-home" (OuterVolumeSpecName: "home") pod "1b606a08-fe7b-407b-8d0f-195184e25ec9" (UID: "1b606a08-fe7b-407b-8d0f-195184e25ec9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:03:48.848160 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.848120 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b606a08-fe7b-407b-8d0f-195184e25ec9" (UID: "1b606a08-fe7b-407b-8d0f-195184e25ec9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:03:48.848160 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.848138 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b606a08-fe7b-407b-8d0f-195184e25ec9" (UID: "1b606a08-fe7b-407b-8d0f-195184e25ec9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:03:48.848272 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.848207 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:48.848272 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.848232 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:48.850013 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.849984 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-dshm" (OuterVolumeSpecName: "dshm") pod "1b606a08-fe7b-407b-8d0f-195184e25ec9" (UID: "1b606a08-fe7b-407b-8d0f-195184e25ec9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:03:48.850127 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.850013 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b606a08-fe7b-407b-8d0f-195184e25ec9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b606a08-fe7b-407b-8d0f-195184e25ec9" (UID: "1b606a08-fe7b-407b-8d0f-195184e25ec9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:03:48.850127 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.850059 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b606a08-fe7b-407b-8d0f-195184e25ec9-kube-api-access-z9t49" (OuterVolumeSpecName: "kube-api-access-z9t49") pod "1b606a08-fe7b-407b-8d0f-195184e25ec9" (UID: "1b606a08-fe7b-407b-8d0f-195184e25ec9"). InnerVolumeSpecName "kube-api-access-z9t49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:03:48.949735 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.949676 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b606a08-fe7b-407b-8d0f-195184e25ec9-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:48.949735 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.949727 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:48.949735 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.949739 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b606a08-fe7b-407b-8d0f-195184e25ec9-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:48.950028 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:48.949753 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9t49\" (UniqueName: \"kubernetes.io/projected/1b606a08-fe7b-407b-8d0f-195184e25ec9-kube-api-access-z9t49\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:49.371172 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.371129 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" podUID="4047b6b9-a339-4f82-8e12-ec46b1bef1a7" containerName="main" containerID="cri-o://9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82" gracePeriod=45 Jul 17 01:03:49.624200 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.624139 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:03:49.644401 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.644355 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:03:49.711697 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.711659 2572 generic.go:356] "Generic (PLEG): container finished" podID="4047b6b9-a339-4f82-8e12-ec46b1bef1a7" containerID="9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82" exitCode=0 Jul 17 01:03:49.711923 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.711720 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" Jul 17 01:03:49.711923 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.711751 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" event={"ID":"4047b6b9-a339-4f82-8e12-ec46b1bef1a7","Type":"ContainerDied","Data":"9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82"} Jul 17 01:03:49.711923 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.711797 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc" event={"ID":"4047b6b9-a339-4f82-8e12-ec46b1bef1a7","Type":"ContainerDied","Data":"dc8710a207a17033fab98d99e402f0a4649aadacb0944cbec268312333241d99"} Jul 17 01:03:49.711923 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.711814 2572 scope.go:117] "RemoveContainer" containerID="9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82" Jul 17 01:03:49.713151 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.713125 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" event={"ID":"1b606a08-fe7b-407b-8d0f-195184e25ec9","Type":"ContainerDied","Data":"bb4b2353226466c34e73362def178848bcf5d67471f34ac203a89823ff9f49e1"} Jul 17 01:03:49.713247 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.713152 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5" Jul 17 01:03:49.720583 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.720564 2572 scope.go:117] "RemoveContainer" containerID="9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82" Jul 17 01:03:49.720863 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:03:49.720841 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82\": container with ID starting with 9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82 not found: ID does not exist" containerID="9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82" Jul 17 01:03:49.720981 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.720872 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82"} err="failed to get container status \"9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82\": rpc error: code = NotFound desc = could not find container \"9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82\": container with ID starting with 9d3424f818c00b6072e9b8900e152ac9539e48fd4a9828d6a97a8f6432c49a82 not found: ID does not exist" Jul 17 01:03:49.720981 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.720911 2572 scope.go:117] "RemoveContainer" containerID="44d69166e8c215b103d360210ea2a7fa6cfd9499967678bef009b495e809f03b" Jul 17 01:03:49.733009 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.732987 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5"] Jul 17 01:03:49.739431 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.739411 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-27f370/test-model-27f370-kserve-cd66ccdc5-wmgf5"] Jul 17 01:03:49.757974 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.757950 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tmp-dir\") pod \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " Jul 17 01:03:49.758074 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.757986 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-model-cache\") pod \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " Jul 17 01:03:49.758074 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758020 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-home\") pod \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " Jul 17 01:03:49.758185 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758080 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-dshm\") pod \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " Jul 17 01:03:49.758185 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758109 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tls-certs\") pod \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " Jul 17 01:03:49.758287 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758194 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw5vx\" (UniqueName: \"kubernetes.io/projected/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-kube-api-access-lw5vx\") pod \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\" (UID: \"4047b6b9-a339-4f82-8e12-ec46b1bef1a7\") " Jul 17 01:03:49.758287 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758210 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4047b6b9-a339-4f82-8e12-ec46b1bef1a7" (UID: "4047b6b9-a339-4f82-8e12-ec46b1bef1a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:03:49.758287 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758224 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "4047b6b9-a339-4f82-8e12-ec46b1bef1a7" (UID: "4047b6b9-a339-4f82-8e12-ec46b1bef1a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:03:49.758416 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758327 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-home" (OuterVolumeSpecName: "home") pod "4047b6b9-a339-4f82-8e12-ec46b1bef1a7" (UID: "4047b6b9-a339-4f82-8e12-ec46b1bef1a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:03:49.758478 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758462 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:49.758532 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758478 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:49.758532 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.758488 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:49.760301 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.760271 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-dshm" (OuterVolumeSpecName: "dshm") pod "4047b6b9-a339-4f82-8e12-ec46b1bef1a7" (UID: "4047b6b9-a339-4f82-8e12-ec46b1bef1a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:03:49.760396 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.760298 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4047b6b9-a339-4f82-8e12-ec46b1bef1a7" (UID: "4047b6b9-a339-4f82-8e12-ec46b1bef1a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:03:49.760396 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.760349 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-kube-api-access-lw5vx" (OuterVolumeSpecName: "kube-api-access-lw5vx") pod "4047b6b9-a339-4f82-8e12-ec46b1bef1a7" (UID: "4047b6b9-a339-4f82-8e12-ec46b1bef1a7"). InnerVolumeSpecName "kube-api-access-lw5vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:03:49.859167 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.859133 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:49.859167 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.859161 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:49.859167 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:49.859174 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw5vx\" (UniqueName: \"kubernetes.io/projected/4047b6b9-a339-4f82-8e12-ec46b1bef1a7-kube-api-access-lw5vx\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:03:50.032653 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:50.032599 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc"] Jul 17 01:03:50.034730 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:50.034708 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e1e207/test-model-e1e207-kserve-6784874965-hfmnc"] Jul 17 01:03:50.087521 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:50.087493 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b606a08-fe7b-407b-8d0f-195184e25ec9" path="/var/lib/kubelet/pods/1b606a08-fe7b-407b-8d0f-195184e25ec9/volumes" Jul 17 01:03:50.087879 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:50.087865 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4047b6b9-a339-4f82-8e12-ec46b1bef1a7" path="/var/lib/kubelet/pods/4047b6b9-a339-4f82-8e12-ec46b1bef1a7/volumes" Jul 17 01:03:51.895925 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:51.895879 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:03:55.434579 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:55.434547 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:03:55.491944 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:03:55.491912 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:00.396825 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:00.396789 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:00.550099 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:00.550055 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:01.422557 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:01.422525 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:01.522334 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:01.522299 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:11.128173 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:11.128137 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:14.694266 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:14.694231 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:04:15.754051 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:15.754015 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:21.589362 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:21.589300 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:21.689331 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:21.689292 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:26.409463 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:26.409424 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:26.618461 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:26.618423 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:26.960441 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:26.960401 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:27.242783 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:27.242688 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:27.421843 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:27.421804 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:30.101775 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:30.101728 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:30.202344 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:30.202308 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:30.381012 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:30.380915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:31.428411 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:31.428364 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:31.486609 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:31.486573 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:34.306394 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:34.306362 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:34.758833 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:34.758800 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:35.068319 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:35.068226 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:35.531479 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:35.531436 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:37.194812 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:37.194779 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:04:37.852365 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:37.852316 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:37.956680 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:37.956636 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:38.050999 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:38.050966 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:38.536222 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:38.536187 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:38.631435 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:38.631397 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:42.325247 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:42.325215 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:43.383021 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.382983 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w"] Jul 17 01:04:43.383397 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.383339 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b606a08-fe7b-407b-8d0f-195184e25ec9" containerName="main" Jul 17 01:04:43.383397 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.383349 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b606a08-fe7b-407b-8d0f-195184e25ec9" containerName="main" Jul 17 01:04:43.383397 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.383364 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4047b6b9-a339-4f82-8e12-ec46b1bef1a7" containerName="main" Jul 17 01:04:43.383397 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.383370 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4047b6b9-a339-4f82-8e12-ec46b1bef1a7" containerName="main" Jul 17 01:04:43.383528 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.383421 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4047b6b9-a339-4f82-8e12-ec46b1bef1a7" containerName="main" Jul 17 01:04:43.383528 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.383431 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b606a08-fe7b-407b-8d0f-195184e25ec9" containerName="main" Jul 17 01:04:43.386530 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.386512 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" Jul 17 01:04:43.388854 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.388821 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:04:43.388976 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.388876 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\"" Jul 17 01:04:43.392296 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.392268 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w"] Jul 17 01:04:43.439792 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.439759 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e192db4-26a1-4509-8d11-bfd1cb11bb3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w\" (UID: \"5e192db4-26a1-4509-8d11-bfd1cb11bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" Jul 17 01:04:43.541087 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.541051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e192db4-26a1-4509-8d11-bfd1cb11bb3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w\" (UID: \"5e192db4-26a1-4509-8d11-bfd1cb11bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" Jul 17 01:04:43.541663 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.541646 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e192db4-26a1-4509-8d11-bfd1cb11bb3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w\" (UID: \"5e192db4-26a1-4509-8d11-bfd1cb11bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" Jul 17 01:04:43.698743 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.698650 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" Jul 17 01:04:43.827761 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.827722 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w"] Jul 17 01:04:43.830267 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:04:43.830240 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e192db4_26a1_4509_8d11_bfd1cb11bb3e.slice/crio-26f5a965e62f0411512a79e31762468179b26afd9faf4dd581bdd91fcbb8b874 WatchSource:0}: Error finding container 26f5a965e62f0411512a79e31762468179b26afd9faf4dd581bdd91fcbb8b874: Status 404 returned error can't find the container with id 26f5a965e62f0411512a79e31762468179b26afd9faf4dd581bdd91fcbb8b874 Jul 17 01:04:43.831952 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.831935 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 01:04:43.960521 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:43.960441 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" event={"ID":"5e192db4-26a1-4509-8d11-bfd1cb11bb3e","Type":"ContainerStarted","Data":"26f5a965e62f0411512a79e31762468179b26afd9faf4dd581bdd91fcbb8b874"} Jul 17 01:04:46.977227 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:46.977192 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerID="70c9a31e02702c0a20947bfbddc2e710046a7f15c7650e9e28b62d53da397ade" exitCode=6 Jul 17 01:04:46.977628 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:46.977261 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" event={"ID":"5e192db4-26a1-4509-8d11-bfd1cb11bb3e","Type":"ContainerDied","Data":"70c9a31e02702c0a20947bfbddc2e710046a7f15c7650e9e28b62d53da397ade"} Jul 17 01:04:46.977628 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:46.977406 2572 scope.go:117] "RemoveContainer" containerID="70c9a31e02702c0a20947bfbddc2e710046a7f15c7650e9e28b62d53da397ade" Jul 17 01:04:47.983511 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:47.983422 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerID="a33497de377fc3ceca015438157134bd76a6b1de398214f1b7e8955bf57adaf7" exitCode=6 Jul 17 01:04:47.983511 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:47.983493 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" event={"ID":"5e192db4-26a1-4509-8d11-bfd1cb11bb3e","Type":"ContainerDied","Data":"a33497de377fc3ceca015438157134bd76a6b1de398214f1b7e8955bf57adaf7"} Jul 17 01:04:47.983986 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:47.983547 2572 scope.go:117] "RemoveContainer" containerID="70c9a31e02702c0a20947bfbddc2e710046a7f15c7650e9e28b62d53da397ade" Jul 17 01:04:47.983986 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:47.983737 2572 scope.go:117] "RemoveContainer" containerID="a33497de377fc3ceca015438157134bd76a6b1de398214f1b7e8955bf57adaf7" Jul 17 01:04:47.983986 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:04:47.983966 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w_odh-ai-gateway-infra(5e192db4-26a1-4509-8d11-bfd1cb11bb3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" Jul 17 01:04:48.990716 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:48.990686 2572 scope.go:117] "RemoveContainer" containerID="a33497de377fc3ceca015438157134bd76a6b1de398214f1b7e8955bf57adaf7" Jul 17 01:04:48.991182 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:04:48.990924 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w_odh-ai-gateway-infra(5e192db4-26a1-4509-8d11-bfd1cb11bb3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" Jul 17 01:04:50.222716 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.222680 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj"] Jul 17 01:04:50.227355 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.227335 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" Jul 17 01:04:50.231104 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.231078 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj"] Jul 17 01:04:50.289937 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.289880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540c3c74-d338-4d5e-a97c-491848138326-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj\" (UID: \"540c3c74-d338-4d5e-a97c-491848138326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" Jul 17 01:04:50.390989 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.390955 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540c3c74-d338-4d5e-a97c-491848138326-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj\" (UID: \"540c3c74-d338-4d5e-a97c-491848138326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" Jul 17 01:04:50.391560 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.391539 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540c3c74-d338-4d5e-a97c-491848138326-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj\" (UID: \"540c3c74-d338-4d5e-a97c-491848138326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" Jul 17 01:04:50.539416 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.539334 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" Jul 17 01:04:50.667207 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.667173 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj"] Jul 17 01:04:50.670592 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:04:50.670568 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540c3c74_d338_4d5e_a97c_491848138326.slice/crio-ccd81b039f4e243e87c3ecb54d13d26a137851d1ac2403864682e226396e1a8d WatchSource:0}: Error finding container ccd81b039f4e243e87c3ecb54d13d26a137851d1ac2403864682e226396e1a8d: Status 404 returned error can't find the container with id ccd81b039f4e243e87c3ecb54d13d26a137851d1ac2403864682e226396e1a8d Jul 17 01:04:50.747550 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.747517 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:50.850668 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:50.850636 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:51.001036 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:51.000993 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" event={"ID":"540c3c74-d338-4d5e-a97c-491848138326","Type":"ContainerStarted","Data":"6e36dd5ab48577958050e7c6f7a047c7131063593c56d285937c295edd271612"} Jul 17 01:04:51.001251 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:51.001057 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" event={"ID":"540c3c74-d338-4d5e-a97c-491848138326","Type":"ContainerStarted","Data":"ccd81b039f4e243e87c3ecb54d13d26a137851d1ac2403864682e226396e1a8d"} Jul 17 01:04:51.015776 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:51.015729 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" podStartSLOduration=1.015713237 podStartE2EDuration="1.015713237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:04:51.014423526 +0000 UTC m=+2863.760932560" watchObservedRunningTime="2026-07-17 01:04:51.015713237 +0000 UTC m=+2863.762222270" Jul 17 01:04:52.007260 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:52.007229 2572 generic.go:356] "Generic (PLEG): container finished" podID="540c3c74-d338-4d5e-a97c-491848138326" containerID="6e36dd5ab48577958050e7c6f7a047c7131063593c56d285937c295edd271612" exitCode=0 Jul 17 01:04:52.007658 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:52.007314 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" event={"ID":"540c3c74-d338-4d5e-a97c-491848138326","Type":"ContainerDied","Data":"6e36dd5ab48577958050e7c6f7a047c7131063593c56d285937c295edd271612"} Jul 17 01:04:53.145956 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:53.145934 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" Jul 17 01:04:53.211137 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:53.211100 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540c3c74-d338-4d5e-a97c-491848138326-maas-api-service-ca\") pod \"540c3c74-d338-4d5e-a97c-491848138326\" (UID: \"540c3c74-d338-4d5e-a97c-491848138326\") " Jul 17 01:04:53.211462 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:53.211434 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540c3c74-d338-4d5e-a97c-491848138326-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "540c3c74-d338-4d5e-a97c-491848138326" (UID: "540c3c74-d338-4d5e-a97c-491848138326"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:04:53.312040 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:53.311953 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540c3c74-d338-4d5e-a97c-491848138326-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:04:54.017115 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:54.017087 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" Jul 17 01:04:54.017289 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:54.017087 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj" event={"ID":"540c3c74-d338-4d5e-a97c-491848138326","Type":"ContainerDied","Data":"ccd81b039f4e243e87c3ecb54d13d26a137851d1ac2403864682e226396e1a8d"} Jul 17 01:04:54.017289 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:54.017207 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccd81b039f4e243e87c3ecb54d13d26a137851d1ac2403864682e226396e1a8d" Jul 17 01:04:54.698912 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:54.698859 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:04:59.278331 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:59.278243 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:04:59.381737 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:04:59.381704 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:00.800804 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:00.800757 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:00.901581 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:00.901533 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:00.962217 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:00.962182 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7"] Jul 17 01:05:00.962566 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:00.962554 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="540c3c74-d338-4d5e-a97c-491848138326" containerName="revoke-keys" Jul 17 01:05:00.962608 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:00.962567 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="540c3c74-d338-4d5e-a97c-491848138326" containerName="revoke-keys" Jul 17 01:05:00.962651 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:00.962643 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="540c3c74-d338-4d5e-a97c-491848138326" containerName="revoke-keys" Jul 17 01:05:00.970659 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:00.970627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" Jul 17 01:05:00.972469 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:00.972359 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7"] Jul 17 01:05:01.078958 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:01.078854 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62545359-076e-41a7-9760-00ab422ed98b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7\" (UID: \"62545359-076e-41a7-9760-00ab422ed98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" Jul 17 01:05:01.093797 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:01.093745 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:01.180025 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:01.179983 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62545359-076e-41a7-9760-00ab422ed98b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7\" (UID: \"62545359-076e-41a7-9760-00ab422ed98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" Jul 17 01:05:01.180627 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:01.180608 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62545359-076e-41a7-9760-00ab422ed98b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7\" (UID: \"62545359-076e-41a7-9760-00ab422ed98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" Jul 17 01:05:01.285209 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:01.285170 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" Jul 17 01:05:01.418730 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:01.418706 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7"] Jul 17 01:05:01.420599 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:05:01.420573 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62545359_076e_41a7_9760_00ab422ed98b.slice/crio-c986a17c9bbd7c8ad079d60c746e0862f918c84b4f77ddb24742c4d4f5846a4c WatchSource:0}: Error finding container c986a17c9bbd7c8ad079d60c746e0862f918c84b4f77ddb24742c4d4f5846a4c: Status 404 returned error can't find the container with id c986a17c9bbd7c8ad079d60c746e0862f918c84b4f77ddb24742c4d4f5846a4c Jul 17 01:05:02.065906 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:02.065865 2572 generic.go:356] "Generic (PLEG): container finished" podID="62545359-076e-41a7-9760-00ab422ed98b" containerID="e9c545218a03f15773cd820e1dbdcc98d78c71a243bfd189510df484cb6b9f0e" exitCode=6 Jul 17 01:05:02.066332 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:02.065953 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" event={"ID":"62545359-076e-41a7-9760-00ab422ed98b","Type":"ContainerDied","Data":"e9c545218a03f15773cd820e1dbdcc98d78c71a243bfd189510df484cb6b9f0e"} Jul 17 01:05:02.066332 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:02.066008 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" event={"ID":"62545359-076e-41a7-9760-00ab422ed98b","Type":"ContainerStarted","Data":"c986a17c9bbd7c8ad079d60c746e0862f918c84b4f77ddb24742c4d4f5846a4c"} Jul 17 01:05:02.066332 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:02.066128 2572 scope.go:117] "RemoveContainer" containerID="e9c545218a03f15773cd820e1dbdcc98d78c71a243bfd189510df484cb6b9f0e" Jul 17 01:05:03.072502 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:03.072421 2572 generic.go:356] "Generic (PLEG): container finished" podID="62545359-076e-41a7-9760-00ab422ed98b" containerID="7386b5df5b5eaeac1d370e2434f8e518fb731634f0f2403665d031f22b9e4071" exitCode=6 Jul 17 01:05:03.072969 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:03.072511 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" event={"ID":"62545359-076e-41a7-9760-00ab422ed98b","Type":"ContainerDied","Data":"7386b5df5b5eaeac1d370e2434f8e518fb731634f0f2403665d031f22b9e4071"} Jul 17 01:05:03.072969 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:03.072574 2572 scope.go:117] "RemoveContainer" containerID="e9c545218a03f15773cd820e1dbdcc98d78c71a243bfd189510df484cb6b9f0e" Jul 17 01:05:03.072969 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:03.072847 2572 scope.go:117] "RemoveContainer" containerID="7386b5df5b5eaeac1d370e2434f8e518fb731634f0f2403665d031f22b9e4071" Jul 17 01:05:03.073104 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:05:03.073081 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7_odh-ai-gateway-infra(62545359-076e-41a7-9760-00ab422ed98b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" podUID="62545359-076e-41a7-9760-00ab422ed98b" Jul 17 01:05:04.079348 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:04.079320 2572 scope.go:117] "RemoveContainer" containerID="7386b5df5b5eaeac1d370e2434f8e518fb731634f0f2403665d031f22b9e4071" Jul 17 01:05:04.079701 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:05:04.079519 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7_odh-ai-gateway-infra(62545359-076e-41a7-9760-00ab422ed98b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" podUID="62545359-076e-41a7-9760-00ab422ed98b" Jul 17 01:05:04.081736 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:04.081720 2572 scope.go:117] "RemoveContainer" containerID="a33497de377fc3ceca015438157134bd76a6b1de398214f1b7e8955bf57adaf7" Jul 17 01:05:05.085875 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:05.085783 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerID="069d5994c42341559e8ca0fb6d3ae96830ca003a1fc9fb9fed29e07788a12c77" exitCode=6 Jul 17 01:05:05.085875 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:05.085820 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" event={"ID":"5e192db4-26a1-4509-8d11-bfd1cb11bb3e","Type":"ContainerDied","Data":"069d5994c42341559e8ca0fb6d3ae96830ca003a1fc9fb9fed29e07788a12c77"} Jul 17 01:05:05.085875 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:05.085874 2572 scope.go:117] "RemoveContainer" containerID="a33497de377fc3ceca015438157134bd76a6b1de398214f1b7e8955bf57adaf7" Jul 17 01:05:05.086411 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:05.086119 2572 scope.go:117] "RemoveContainer" containerID="069d5994c42341559e8ca0fb6d3ae96830ca003a1fc9fb9fed29e07788a12c77" Jul 17 01:05:05.086411 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:05:05.086319 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w_odh-ai-gateway-infra(5e192db4-26a1-4509-8d11-bfd1cb11bb3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" Jul 17 01:05:05.195939 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:05.195905 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:05:05.312717 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:05.312683 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:06.109441 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:06.109410 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w"] Jul 17 01:05:06.245314 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:06.245291 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" Jul 17 01:05:06.328675 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:06.328644 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e192db4-26a1-4509-8d11-bfd1cb11bb3e-maas-api-service-ca\") pod \"5e192db4-26a1-4509-8d11-bfd1cb11bb3e\" (UID: \"5e192db4-26a1-4509-8d11-bfd1cb11bb3e\") " Jul 17 01:05:06.329069 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:06.329037 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e192db4-26a1-4509-8d11-bfd1cb11bb3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e192db4-26a1-4509-8d11-bfd1cb11bb3e" (UID: "5e192db4-26a1-4509-8d11-bfd1cb11bb3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:05:06.429734 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:06.429653 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e192db4-26a1-4509-8d11-bfd1cb11bb3e-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:05:07.099905 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:07.099852 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" event={"ID":"5e192db4-26a1-4509-8d11-bfd1cb11bb3e","Type":"ContainerDied","Data":"26f5a965e62f0411512a79e31762468179b26afd9faf4dd581bdd91fcbb8b874"} Jul 17 01:05:07.099905 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:07.099881 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w" Jul 17 01:05:07.100154 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:07.099931 2572 scope.go:117] "RemoveContainer" containerID="069d5994c42341559e8ca0fb6d3ae96830ca003a1fc9fb9fed29e07788a12c77" Jul 17 01:05:07.123165 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:07.123140 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w"] Jul 17 01:05:07.130376 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:07.130350 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-89p9w"] Jul 17 01:05:08.088551 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:08.088519 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" path="/var/lib/kubelet/pods/5e192db4-26a1-4509-8d11-bfd1cb11bb3e/volumes" Jul 17 01:05:10.904647 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:10.904604 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:10.974970 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:10.974931 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:15.081990 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:15.081957 2572 scope.go:117] "RemoveContainer" containerID="7386b5df5b5eaeac1d370e2434f8e518fb731634f0f2403665d031f22b9e4071" Jul 17 01:05:15.961074 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:15.960985 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:16.142486 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:16.142454 2572 generic.go:356] "Generic (PLEG): container finished" podID="62545359-076e-41a7-9760-00ab422ed98b" containerID="ec99985c6d823c9dcb1a92e709974998ff5f501669ebf27ab55877ecce1abd7b" exitCode=6 Jul 17 01:05:16.142959 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:16.142515 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" event={"ID":"62545359-076e-41a7-9760-00ab422ed98b","Type":"ContainerDied","Data":"ec99985c6d823c9dcb1a92e709974998ff5f501669ebf27ab55877ecce1abd7b"} Jul 17 01:05:16.142959 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:16.142578 2572 scope.go:117] "RemoveContainer" containerID="7386b5df5b5eaeac1d370e2434f8e518fb731634f0f2403665d031f22b9e4071" Jul 17 01:05:16.142959 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:16.142913 2572 scope.go:117] "RemoveContainer" containerID="ec99985c6d823c9dcb1a92e709974998ff5f501669ebf27ab55877ecce1abd7b" Jul 17 01:05:16.143192 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:05:16.143166 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7_odh-ai-gateway-infra(62545359-076e-41a7-9760-00ab422ed98b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" podUID="62545359-076e-41a7-9760-00ab422ed98b" Jul 17 01:05:16.174663 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:16.174620 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:16.991722 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:16.991685 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:17.165857 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:17.165825 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7"] Jul 17 01:05:17.296968 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:17.296944 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" Jul 17 01:05:17.425831 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:17.425793 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62545359-076e-41a7-9760-00ab422ed98b-maas-api-service-ca\") pod \"62545359-076e-41a7-9760-00ab422ed98b\" (UID: \"62545359-076e-41a7-9760-00ab422ed98b\") " Jul 17 01:05:17.426222 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:17.426199 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62545359-076e-41a7-9760-00ab422ed98b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62545359-076e-41a7-9760-00ab422ed98b" (UID: "62545359-076e-41a7-9760-00ab422ed98b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:05:17.527482 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:17.527395 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62545359-076e-41a7-9760-00ab422ed98b-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:05:17.585398 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:17.585363 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:18.154871 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:18.154842 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" Jul 17 01:05:18.154871 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:18.154867 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7" event={"ID":"62545359-076e-41a7-9760-00ab422ed98b","Type":"ContainerDied","Data":"c986a17c9bbd7c8ad079d60c746e0862f918c84b4f77ddb24742c4d4f5846a4c"} Jul 17 01:05:18.155139 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:18.154939 2572 scope.go:117] "RemoveContainer" containerID="ec99985c6d823c9dcb1a92e709974998ff5f501669ebf27ab55877ecce1abd7b" Jul 17 01:05:18.169838 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:18.169816 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7"] Jul 17 01:05:18.174513 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:18.174488 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-6d4b7"] Jul 17 01:05:20.087622 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:20.087577 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62545359-076e-41a7-9760-00ab422ed98b" path="/var/lib/kubelet/pods/62545359-076e-41a7-9760-00ab422ed98b/volumes" Jul 17 01:05:26.741793 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:26.741756 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:28.058311 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:28.058277 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:28.157812 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:28.157783 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:30.293795 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:30.293761 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:05:31.387632 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:31.387601 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:37.062880 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:37.062847 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:37.163685 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:37.163656 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:42.058742 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:42.058707 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:42.277216 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:42.277166 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:42.610084 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:42.610050 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:44.424673 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:44.424638 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:46.292369 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:46.292332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:46.400677 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:46.400647 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:46.490956 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:46.490910 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:46.590845 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:46.590736 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:47.039024 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:47.038987 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:47.141560 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:47.141525 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:50.712504 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:50.712469 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:51.397581 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:51.397547 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:05:51.586017 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:51.585973 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:51.637641 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:51.637610 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:54.268681 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:54.268644 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:54.370160 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:54.370127 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:54.468180 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:54.468150 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:54.674206 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:54.674166 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:55.020240 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:55.020141 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:55.120461 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:55.120419 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:56.473754 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.473715 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8"] Jul 17 01:05:56.474159 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474124 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerName="revoke-keys" Jul 17 01:05:56.474159 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474137 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerName="revoke-keys" Jul 17 01:05:56.474159 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474145 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerName="revoke-keys" Jul 17 01:05:56.474159 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474151 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerName="revoke-keys" Jul 17 01:05:56.474159 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474160 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62545359-076e-41a7-9760-00ab422ed98b" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474166 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="62545359-076e-41a7-9760-00ab422ed98b" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474172 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62545359-076e-41a7-9760-00ab422ed98b" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474177 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="62545359-076e-41a7-9760-00ab422ed98b" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474185 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62545359-076e-41a7-9760-00ab422ed98b" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474190 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="62545359-076e-41a7-9760-00ab422ed98b" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474198 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474204 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474258 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="62545359-076e-41a7-9760-00ab422ed98b" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474268 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474274 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474279 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="62545359-076e-41a7-9760-00ab422ed98b" containerName="revoke-keys" Jul 17 01:05:56.474324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.474286 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e192db4-26a1-4509-8d11-bfd1cb11bb3e" containerName="revoke-keys" Jul 17 01:05:56.477643 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.477627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" Jul 17 01:05:56.479957 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.479936 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:05:56.480068 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.479936 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\"" Jul 17 01:05:56.483769 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.483730 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8"] Jul 17 01:05:56.556050 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.556012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc9b497-40cd-42a6-906b-363f4eeef831-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-zbxh8\" (UID: \"0dc9b497-40cd-42a6-906b-363f4eeef831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" Jul 17 01:05:56.657196 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.657161 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc9b497-40cd-42a6-906b-363f4eeef831-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-zbxh8\" (UID: \"0dc9b497-40cd-42a6-906b-363f4eeef831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" Jul 17 01:05:56.657732 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.657713 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc9b497-40cd-42a6-906b-363f4eeef831-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-zbxh8\" (UID: \"0dc9b497-40cd-42a6-906b-363f4eeef831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" Jul 17 01:05:56.789536 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.789443 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" Jul 17 01:05:56.918115 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:56.918079 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8"] Jul 17 01:05:56.921229 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:05:56.921192 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc9b497_40cd_42a6_906b_363f4eeef831.slice/crio-c9e9db878dde52492783fc8c925e8ebf90b1d8591f8cffeb51b7e61e7d635ffd WatchSource:0}: Error finding container c9e9db878dde52492783fc8c925e8ebf90b1d8591f8cffeb51b7e61e7d635ffd: Status 404 returned error can't find the container with id c9e9db878dde52492783fc8c925e8ebf90b1d8591f8cffeb51b7e61e7d635ffd Jul 17 01:05:57.348490 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:57.348461 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" event={"ID":"0dc9b497-40cd-42a6-906b-363f4eeef831","Type":"ContainerStarted","Data":"7c75f0032d7be86ef4a2c8fcf5518ed495cf9d81901410732b28fc182414f55c"} Jul 17 01:05:57.348656 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:57.348512 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" event={"ID":"0dc9b497-40cd-42a6-906b-363f4eeef831","Type":"ContainerStarted","Data":"c9e9db878dde52492783fc8c925e8ebf90b1d8591f8cffeb51b7e61e7d635ffd"} Jul 17 01:05:57.362757 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:57.362715 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" podStartSLOduration=1.362700346 podStartE2EDuration="1.362700346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:05:57.361180346 +0000 UTC m=+2930.107689380" watchObservedRunningTime="2026-07-17 01:05:57.362700346 +0000 UTC m=+2930.109209402" Jul 17 01:05:58.355425 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:58.355388 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerID="7c75f0032d7be86ef4a2c8fcf5518ed495cf9d81901410732b28fc182414f55c" exitCode=6 Jul 17 01:05:58.355856 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:58.355476 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" event={"ID":"0dc9b497-40cd-42a6-906b-363f4eeef831","Type":"ContainerDied","Data":"7c75f0032d7be86ef4a2c8fcf5518ed495cf9d81901410732b28fc182414f55c"} Jul 17 01:05:58.355856 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:58.355638 2572 scope.go:117] "RemoveContainer" containerID="7c75f0032d7be86ef4a2c8fcf5518ed495cf9d81901410732b28fc182414f55c" Jul 17 01:05:58.743565 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:58.743532 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:05:59.361949 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:59.361920 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerID="f4280728922b9aaf9490fa8447c1b032db77e704f771c2027a4b9e44239fdba5" exitCode=6 Jul 17 01:05:59.362382 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:59.361972 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" event={"ID":"0dc9b497-40cd-42a6-906b-363f4eeef831","Type":"ContainerDied","Data":"f4280728922b9aaf9490fa8447c1b032db77e704f771c2027a4b9e44239fdba5"} Jul 17 01:05:59.362382 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:59.362019 2572 scope.go:117] "RemoveContainer" containerID="7c75f0032d7be86ef4a2c8fcf5518ed495cf9d81901410732b28fc182414f55c" Jul 17 01:05:59.362382 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:05:59.362195 2572 scope.go:117] "RemoveContainer" containerID="f4280728922b9aaf9490fa8447c1b032db77e704f771c2027a4b9e44239fdba5" Jul 17 01:05:59.362491 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:05:59.362442 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-zbxh8_odh-ai-gateway-infra(0dc9b497-40cd-42a6-906b-363f4eeef831)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" Jul 17 01:06:00.367672 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:00.367646 2572 scope.go:117] "RemoveContainer" containerID="f4280728922b9aaf9490fa8447c1b032db77e704f771c2027a4b9e44239fdba5" Jul 17 01:06:00.368065 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:06:00.367867 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-zbxh8_odh-ai-gateway-infra(0dc9b497-40cd-42a6-906b-363f4eeef831)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" Jul 17 01:06:05.198338 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:05.198295 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:06:07.892690 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:07.892650 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:06:08.002329 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:08.002289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:06:15.081966 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:15.081933 2572 scope.go:117] "RemoveContainer" containerID="f4280728922b9aaf9490fa8447c1b032db77e704f771c2027a4b9e44239fdba5" Jul 17 01:06:15.440696 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:15.440663 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" event={"ID":"0dc9b497-40cd-42a6-906b-363f4eeef831","Type":"ContainerStarted","Data":"6828d78d8e02804449342a38da444faa6bd78a1e18e85e1007dff105fdc048b9"} Jul 17 01:06:15.907041 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:15.907007 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:06:16.008425 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.008329 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:06:16.446553 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.446523 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerID="6828d78d8e02804449342a38da444faa6bd78a1e18e85e1007dff105fdc048b9" exitCode=6 Jul 17 01:06:16.446991 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.446597 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" event={"ID":"0dc9b497-40cd-42a6-906b-363f4eeef831","Type":"ContainerDied","Data":"6828d78d8e02804449342a38da444faa6bd78a1e18e85e1007dff105fdc048b9"} Jul 17 01:06:16.446991 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.446663 2572 scope.go:117] "RemoveContainer" containerID="f4280728922b9aaf9490fa8447c1b032db77e704f771c2027a4b9e44239fdba5" Jul 17 01:06:16.446991 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.446955 2572 scope.go:117] "RemoveContainer" containerID="6828d78d8e02804449342a38da444faa6bd78a1e18e85e1007dff105fdc048b9" Jul 17 01:06:16.447216 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:06:16.447192 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-zbxh8_odh-ai-gateway-infra(0dc9b497-40cd-42a6-906b-363f4eeef831)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" Jul 17 01:06:16.469288 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.469259 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8"] Jul 17 01:06:16.889516 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.889475 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs"] Jul 17 01:06:16.890136 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.890120 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="62545359-076e-41a7-9760-00ab422ed98b" containerName="revoke-keys" Jul 17 01:06:16.894467 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.894448 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:16.898006 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.897937 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e1dd24\"/\"openshift-service-ca.crt\"" Jul 17 01:06:16.898006 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.897998 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e1dd24\"/\"auth-test-model-e1dd24-kserve-self-signed-certs\"" Jul 17 01:06:16.898204 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.898024 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e1dd24\"/\"default-dockercfg-64w99\"" Jul 17 01:06:16.898204 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.898023 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e1dd24\"/\"kube-root-ca.crt\"" Jul 17 01:06:16.900492 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.900471 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs"] Jul 17 01:06:16.915512 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:16.915487 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:06:17.056293 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.056258 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-tmp-dir\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.056503 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.056306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-dshm\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.056503 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.056376 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4m2j\" (UniqueName: \"kubernetes.io/projected/edc515b2-4b5d-43d5-a561-59090a7ddd28-kube-api-access-p4m2j\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.056503 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.056402 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc515b2-4b5d-43d5-a561-59090a7ddd28-tls-certs\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.056503 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.056430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-model-cache\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.056503 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.056448 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-home\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.158106 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.158004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4m2j\" (UniqueName: \"kubernetes.io/projected/edc515b2-4b5d-43d5-a561-59090a7ddd28-kube-api-access-p4m2j\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.158106 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.158071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc515b2-4b5d-43d5-a561-59090a7ddd28-tls-certs\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.158429 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.158125 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-model-cache\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.158429 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.158156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-home\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.158429 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.158231 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-tmp-dir\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.158429 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.158269 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-dshm\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.158629 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.158599 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-model-cache\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.158747 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.158635 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-tmp-dir\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.158747 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.158676 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-home\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.160399 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.160381 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-dshm\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.160814 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.160797 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc515b2-4b5d-43d5-a561-59090a7ddd28-tls-certs\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.165452 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.165430 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4m2j\" (UniqueName: \"kubernetes.io/projected/edc515b2-4b5d-43d5-a561-59090a7ddd28-kube-api-access-p4m2j\") pod \"auth-test-model-e1dd24-kserve-57579669f9-2ldxs\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.204592 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.204555 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:17.336473 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.336266 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs"] Jul 17 01:06:17.338748 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:06:17.338724 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedc515b2_4b5d_43d5_a561_59090a7ddd28.slice/crio-b2af5b314da39ccefecf9d485f3399869bee15e940f242ff28e68f440780caaf WatchSource:0}: Error finding container b2af5b314da39ccefecf9d485f3399869bee15e940f242ff28e68f440780caaf: Status 404 returned error can't find the container with id b2af5b314da39ccefecf9d485f3399869bee15e940f242ff28e68f440780caaf Jul 17 01:06:17.454716 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.454682 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" event={"ID":"edc515b2-4b5d-43d5-a561-59090a7ddd28","Type":"ContainerStarted","Data":"6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae"} Jul 17 01:06:17.455077 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.454730 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" event={"ID":"edc515b2-4b5d-43d5-a561-59090a7ddd28","Type":"ContainerStarted","Data":"b2af5b314da39ccefecf9d485f3399869bee15e940f242ff28e68f440780caaf"} Jul 17 01:06:17.483781 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.483730 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" podStartSLOduration=1.483715139 podStartE2EDuration="1.483715139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:06:17.481975943 +0000 UTC m=+2950.228484978" watchObservedRunningTime="2026-07-17 01:06:17.483715139 +0000 UTC m=+2950.230224172" Jul 17 01:06:17.572648 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.572625 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" Jul 17 01:06:17.663311 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.663262 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc9b497-40cd-42a6-906b-363f4eeef831-maas-api-service-ca\") pod \"0dc9b497-40cd-42a6-906b-363f4eeef831\" (UID: \"0dc9b497-40cd-42a6-906b-363f4eeef831\") " Jul 17 01:06:17.663695 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.663672 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc9b497-40cd-42a6-906b-363f4eeef831-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dc9b497-40cd-42a6-906b-363f4eeef831" (UID: "0dc9b497-40cd-42a6-906b-363f4eeef831"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:06:17.764389 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:17.764303 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc9b497-40cd-42a6-906b-363f4eeef831-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:06:18.460279 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:18.460202 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" Jul 17 01:06:18.460279 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:18.460237 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8" event={"ID":"0dc9b497-40cd-42a6-906b-363f4eeef831","Type":"ContainerDied","Data":"c9e9db878dde52492783fc8c925e8ebf90b1d8591f8cffeb51b7e61e7d635ffd"} Jul 17 01:06:18.460772 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:18.460300 2572 scope.go:117] "RemoveContainer" containerID="6828d78d8e02804449342a38da444faa6bd78a1e18e85e1007dff105fdc048b9" Jul 17 01:06:18.475029 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:18.475006 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8"] Jul 17 01:06:18.478960 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:18.478939 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-zbxh8"] Jul 17 01:06:20.093155 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:20.093118 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" path="/var/lib/kubelet/pods/0dc9b497-40cd-42a6-906b-363f4eeef831/volumes" Jul 17 01:06:20.597249 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:20.597217 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:06:23.128718 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:23.128685 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:06:27.205494 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:27.205458 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:27.205953 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:27.205511 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:27.217708 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:27.217687 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:27.513748 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:27.513662 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:06:41.050174 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.050097 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x"] Jul 17 01:06:41.050518 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.050439 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerName="revoke-keys" Jul 17 01:06:41.050518 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.050450 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerName="revoke-keys" Jul 17 01:06:41.050518 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.050466 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerName="revoke-keys" Jul 17 01:06:41.050518 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.050472 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerName="revoke-keys" Jul 17 01:06:41.050518 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.050480 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerName="revoke-keys" Jul 17 01:06:41.050518 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.050487 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerName="revoke-keys" Jul 17 01:06:41.050708 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.050543 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerName="revoke-keys" Jul 17 01:06:41.050708 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.050553 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerName="revoke-keys" Jul 17 01:06:41.054703 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.054687 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" Jul 17 01:06:41.057723 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.057695 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:06:41.058659 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.058635 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\"" Jul 17 01:06:41.059525 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.059498 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x"] Jul 17 01:06:41.169350 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.169321 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc462517-6713-4988-bd31-fb243e8ad5eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x\" (UID: \"fc462517-6713-4988-bd31-fb243e8ad5eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" Jul 17 01:06:41.270158 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.270117 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc462517-6713-4988-bd31-fb243e8ad5eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x\" (UID: \"fc462517-6713-4988-bd31-fb243e8ad5eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" Jul 17 01:06:41.270702 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.270682 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc462517-6713-4988-bd31-fb243e8ad5eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x\" (UID: \"fc462517-6713-4988-bd31-fb243e8ad5eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" Jul 17 01:06:41.366027 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.365985 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" Jul 17 01:06:41.491221 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.491200 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x"] Jul 17 01:06:41.568797 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:41.568767 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" event={"ID":"fc462517-6713-4988-bd31-fb243e8ad5eb","Type":"ContainerStarted","Data":"1a0ac2a85700e99341399ac097a8a71f704fc594b4fbab80d75228bc79c4dd96"} Jul 17 01:06:42.575033 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.574950 2572 generic.go:356] "Generic (PLEG): container finished" podID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerID="234c75848e273b3d2aa80c5abea6167573e0c0ebad05a84282f241a4d38565ef" exitCode=6 Jul 17 01:06:42.575033 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.575008 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" event={"ID":"fc462517-6713-4988-bd31-fb243e8ad5eb","Type":"ContainerDied","Data":"234c75848e273b3d2aa80c5abea6167573e0c0ebad05a84282f241a4d38565ef"} Jul 17 01:06:42.575432 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.575208 2572 scope.go:117] "RemoveContainer" containerID="234c75848e273b3d2aa80c5abea6167573e0c0ebad05a84282f241a4d38565ef" Jul 17 01:06:42.884297 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.884265 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g"] Jul 17 01:06:42.884700 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.884687 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc9b497-40cd-42a6-906b-363f4eeef831" containerName="revoke-keys" Jul 17 01:06:42.887615 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.887598 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:42.891414 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.891392 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"openshift-service-ca.crt\"" Jul 17 01:06:42.891729 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.891700 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"kube-root-ca.crt\"" Jul 17 01:06:42.891819 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.891784 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"auth-test-model-ac54e2-kserve-self-signed-certs\"" Jul 17 01:06:42.891877 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.891824 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"default-dockercfg-nhtws\"" Jul 17 01:06:42.898288 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.898262 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g"] Jul 17 01:06:42.906419 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.906390 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:06:42.986225 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.986187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-dshm\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:42.986389 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.986254 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-home\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:42.986389 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.986289 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tls-certs\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:42.986389 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.986335 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tmp-dir\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:42.986522 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.986395 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-model-cache\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:42.986522 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:42.986459 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6fp4\" (UniqueName: \"kubernetes.io/projected/2f74753a-1523-4f60-8950-e5e6dd9b3b49-kube-api-access-g6fp4\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.087124 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.087091 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-model-cache\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.087124 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.087127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6fp4\" (UniqueName: \"kubernetes.io/projected/2f74753a-1523-4f60-8950-e5e6dd9b3b49-kube-api-access-g6fp4\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.087356 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.087149 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-dshm\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.087356 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.087182 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-home\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.087356 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.087198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tls-certs\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.087356 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.087328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tmp-dir\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.087582 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.087544 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-model-cache\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.087652 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.087595 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-home\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.087702 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.087666 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tmp-dir\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.089470 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.089448 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-dshm\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.089664 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.089647 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tls-certs\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.094856 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.094836 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6fp4\" (UniqueName: \"kubernetes.io/projected/2f74753a-1523-4f60-8950-e5e6dd9b3b49-kube-api-access-g6fp4\") pod \"auth-test-model-ac54e2-kserve-849778ff5-9vr9g\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.204337 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.204309 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:43.330163 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.330133 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g"] Jul 17 01:06:43.331499 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:06:43.331467 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f74753a_1523_4f60_8950_e5e6dd9b3b49.slice/crio-79e75e7f7b933b9666f299a5100f3c923a96f4470ead48f472b4f47f0b4ad25c WatchSource:0}: Error finding container 79e75e7f7b933b9666f299a5100f3c923a96f4470ead48f472b4f47f0b4ad25c: Status 404 returned error can't find the container with id 79e75e7f7b933b9666f299a5100f3c923a96f4470ead48f472b4f47f0b4ad25c Jul 17 01:06:43.580666 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.580569 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" event={"ID":"2f74753a-1523-4f60-8950-e5e6dd9b3b49","Type":"ContainerStarted","Data":"fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c"} Jul 17 01:06:43.580666 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.580614 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" event={"ID":"2f74753a-1523-4f60-8950-e5e6dd9b3b49","Type":"ContainerStarted","Data":"79e75e7f7b933b9666f299a5100f3c923a96f4470ead48f472b4f47f0b4ad25c"} Jul 17 01:06:43.582425 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.582397 2572 generic.go:356] "Generic (PLEG): container finished" podID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerID="b7993c8eb0b58786de1e4158c258f9321144bdf5cbf700e5e8c6601c8160559f" exitCode=6 Jul 17 01:06:43.582542 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.582477 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" event={"ID":"fc462517-6713-4988-bd31-fb243e8ad5eb","Type":"ContainerDied","Data":"b7993c8eb0b58786de1e4158c258f9321144bdf5cbf700e5e8c6601c8160559f"} Jul 17 01:06:43.582595 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.582558 2572 scope.go:117] "RemoveContainer" containerID="234c75848e273b3d2aa80c5abea6167573e0c0ebad05a84282f241a4d38565ef" Jul 17 01:06:43.582635 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.582618 2572 scope.go:117] "RemoveContainer" containerID="b7993c8eb0b58786de1e4158c258f9321144bdf5cbf700e5e8c6601c8160559f" Jul 17 01:06:43.582812 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:06:43.582795 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x_odh-ai-gateway-infra(fc462517-6713-4988-bd31-fb243e8ad5eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" Jul 17 01:06:43.596574 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:43.596192 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" podStartSLOduration=1.596176233 podStartE2EDuration="1.596176233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:06:43.595570263 +0000 UTC m=+2976.342079297" watchObservedRunningTime="2026-07-17 01:06:43.596176233 +0000 UTC m=+2976.342685268" Jul 17 01:06:44.589211 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:44.589185 2572 scope.go:117] "RemoveContainer" containerID="b7993c8eb0b58786de1e4158c258f9321144bdf5cbf700e5e8c6601c8160559f" Jul 17 01:06:44.589648 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:06:44.589380 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x_odh-ai-gateway-infra(fc462517-6713-4988-bd31-fb243e8ad5eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" Jul 17 01:06:46.795494 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:46.795455 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:06:53.205099 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:53.205057 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:53.205099 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:53.205106 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:53.217692 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:53.217669 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:53.643293 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:53.643266 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:06:57.082428 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:57.082389 2572 scope.go:117] "RemoveContainer" containerID="b7993c8eb0b58786de1e4158c258f9321144bdf5cbf700e5e8c6601c8160559f" Jul 17 01:06:57.654310 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:57.654234 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" event={"ID":"fc462517-6713-4988-bd31-fb243e8ad5eb","Type":"ContainerStarted","Data":"63fb7672add688be774aca417b3e62f39f50e80845064fad91b865dc19fdfa69"} Jul 17 01:06:57.669089 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:57.669033 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" podStartSLOduration=16.669019098 podStartE2EDuration="16.669019098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:06:57.667842623 +0000 UTC m=+2990.414351657" watchObservedRunningTime="2026-07-17 01:06:57.669019098 +0000 UTC m=+2990.415528131" Jul 17 01:06:58.660583 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:58.660549 2572 generic.go:356] "Generic (PLEG): container finished" podID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerID="63fb7672add688be774aca417b3e62f39f50e80845064fad91b865dc19fdfa69" exitCode=6 Jul 17 01:06:58.661043 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:58.660622 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" event={"ID":"fc462517-6713-4988-bd31-fb243e8ad5eb","Type":"ContainerDied","Data":"63fb7672add688be774aca417b3e62f39f50e80845064fad91b865dc19fdfa69"} Jul 17 01:06:58.661043 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:58.660679 2572 scope.go:117] "RemoveContainer" containerID="b7993c8eb0b58786de1e4158c258f9321144bdf5cbf700e5e8c6601c8160559f" Jul 17 01:06:58.661043 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:58.660939 2572 scope.go:117] "RemoveContainer" containerID="63fb7672add688be774aca417b3e62f39f50e80845064fad91b865dc19fdfa69" Jul 17 01:06:58.661178 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:06:58.661112 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x_odh-ai-gateway-infra(fc462517-6713-4988-bd31-fb243e8ad5eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" Jul 17 01:06:58.683960 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:58.683930 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x"] Jul 17 01:06:59.800113 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:59.800091 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" Jul 17 01:06:59.936116 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:59.936049 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc462517-6713-4988-bd31-fb243e8ad5eb-maas-api-service-ca\") pod \"fc462517-6713-4988-bd31-fb243e8ad5eb\" (UID: \"fc462517-6713-4988-bd31-fb243e8ad5eb\") " Jul 17 01:06:59.936412 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:06:59.936389 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc462517-6713-4988-bd31-fb243e8ad5eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc462517-6713-4988-bd31-fb243e8ad5eb" (UID: "fc462517-6713-4988-bd31-fb243e8ad5eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:07:00.036923 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:00.036880 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc462517-6713-4988-bd31-fb243e8ad5eb-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:07:00.597958 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:00.597919 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:07:00.672535 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:00.672506 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" Jul 17 01:07:00.672688 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:00.672549 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x" event={"ID":"fc462517-6713-4988-bd31-fb243e8ad5eb","Type":"ContainerDied","Data":"1a0ac2a85700e99341399ac097a8a71f704fc594b4fbab80d75228bc79c4dd96"} Jul 17 01:07:00.672688 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:00.672599 2572 scope.go:117] "RemoveContainer" containerID="63fb7672add688be774aca417b3e62f39f50e80845064fad91b865dc19fdfa69" Jul 17 01:07:00.689866 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:00.689846 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x"] Jul 17 01:07:00.699397 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:00.699377 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-hlr9x"] Jul 17 01:07:02.087282 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:02.087248 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" path="/var/lib/kubelet/pods/fc462517-6713-4988-bd31-fb243e8ad5eb/volumes" Jul 17 01:07:19.289612 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:19.289576 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:07:22.471829 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:22.471789 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:07:22.530562 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:22.530529 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:07:26.371850 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:26.371809 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:07:34.592564 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:34.592524 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:07:41.502463 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:41.502425 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:07:52.788758 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:07:52.788726 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:08:10.185981 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:10.185901 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:08:28.486425 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:28.486387 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:08:43.835938 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:43.835900 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs"] Jul 17 01:08:43.854799 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:43.854758 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:08:44.715199 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:44.715171 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g"] Jul 17 01:08:44.733170 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:44.733137 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:08:48.787257 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:48.787229 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:08:53.096062 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.096033 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks"] Jul 17 01:08:53.096450 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.096436 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerName="revoke-keys" Jul 17 01:08:53.096499 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.096451 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerName="revoke-keys" Jul 17 01:08:53.096499 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.096459 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerName="revoke-keys" Jul 17 01:08:53.096499 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.096466 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerName="revoke-keys" Jul 17 01:08:53.096600 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.096552 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerName="revoke-keys" Jul 17 01:08:53.096600 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.096564 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerName="revoke-keys" Jul 17 01:08:53.096600 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.096571 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerName="revoke-keys" Jul 17 01:08:53.099424 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.099409 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.101981 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.101957 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e1dd24\"/\"sub-test-model-e1dd24-kserve-self-signed-certs\"" Jul 17 01:08:53.108444 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.108421 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks"] Jul 17 01:08:53.119568 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.119193 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:08:53.279954 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.279932 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-model-cache\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.280075 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.279960 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38781528-cb08-4082-a5dc-6cf8a1e974da-tls-certs\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.280075 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.279979 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-home\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.280148 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.280086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-dshm\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.280148 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.280109 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-tmp-dir\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.280148 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.280139 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxqv\" (UniqueName: \"kubernetes.io/projected/38781528-cb08-4082-a5dc-6cf8a1e974da-kube-api-access-dgxqv\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.380730 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.380661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxqv\" (UniqueName: \"kubernetes.io/projected/38781528-cb08-4082-a5dc-6cf8a1e974da-kube-api-access-dgxqv\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.380730 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.380712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-model-cache\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.380730 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.380730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38781528-cb08-4082-a5dc-6cf8a1e974da-tls-certs\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.380966 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.380787 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-home\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.380966 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.380910 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-dshm\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.380966 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.380945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-tmp-dir\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.381197 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.381179 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-model-cache\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.381269 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.381255 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-tmp-dir\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.381315 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.381253 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-home\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.383032 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.383012 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-dshm\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.383234 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.383214 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38781528-cb08-4082-a5dc-6cf8a1e974da-tls-certs\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.388655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.388630 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxqv\" (UniqueName: \"kubernetes.io/projected/38781528-cb08-4082-a5dc-6cf8a1e974da-kube-api-access-dgxqv\") pod \"sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.408706 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.408682 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:08:53.536967 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:53.536935 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks"] Jul 17 01:08:53.540101 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:08:53.540072 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38781528_cb08_4082_a5dc_6cf8a1e974da.slice/crio-5096762431ca3de23f13141cfbfc2679a57539fe68d458a54643980eb41d2ade WatchSource:0}: Error finding container 5096762431ca3de23f13141cfbfc2679a57539fe68d458a54643980eb41d2ade: Status 404 returned error can't find the container with id 5096762431ca3de23f13141cfbfc2679a57539fe68d458a54643980eb41d2ade Jul 17 01:08:54.190632 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:54.190597 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" event={"ID":"38781528-cb08-4082-a5dc-6cf8a1e974da","Type":"ContainerStarted","Data":"b0060c81c423a6001da00666310860835edd85e88239af65bc5cdfe7ec0cfd3b"} Jul 17 01:08:54.191034 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:54.190641 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" event={"ID":"38781528-cb08-4082-a5dc-6cf8a1e974da","Type":"ContainerStarted","Data":"5096762431ca3de23f13141cfbfc2679a57539fe68d458a54643980eb41d2ade"} Jul 17 01:08:54.207045 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:54.206999 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" podStartSLOduration=1.206984054 podStartE2EDuration="1.206984054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:08:54.205309465 +0000 UTC m=+3106.951818502" watchObservedRunningTime="2026-07-17 01:08:54.206984054 +0000 UTC m=+3106.953493087" Jul 17 01:08:58.857789 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:58.857750 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" podUID="edc515b2-4b5d-43d5-a561-59090a7ddd28" containerName="main" containerID="cri-o://6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae" gracePeriod=45 Jul 17 01:08:59.105984 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.105963 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:08:59.215258 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.215183 2572 generic.go:356] "Generic (PLEG): container finished" podID="edc515b2-4b5d-43d5-a561-59090a7ddd28" containerID="6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae" exitCode=0 Jul 17 01:08:59.215258 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.215225 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" Jul 17 01:08:59.215424 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.215265 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" event={"ID":"edc515b2-4b5d-43d5-a561-59090a7ddd28","Type":"ContainerDied","Data":"6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae"} Jul 17 01:08:59.215424 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.215307 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs" event={"ID":"edc515b2-4b5d-43d5-a561-59090a7ddd28","Type":"ContainerDied","Data":"b2af5b314da39ccefecf9d485f3399869bee15e940f242ff28e68f440780caaf"} Jul 17 01:08:59.215424 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.215325 2572 scope.go:117] "RemoveContainer" containerID="6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae" Jul 17 01:08:59.223481 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.223464 2572 scope.go:117] "RemoveContainer" containerID="6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae" Jul 17 01:08:59.223703 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:08:59.223686 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae\": container with ID starting with 6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae not found: ID does not exist" containerID="6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae" Jul 17 01:08:59.223749 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.223714 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae"} err="failed to get container status \"6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae\": rpc error: code = NotFound desc = could not find container \"6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae\": container with ID starting with 6e392031074751a8cb7f82f76a8819f3d59577bcda3f112636096e9b432594ae not found: ID does not exist" Jul 17 01:08:59.227006 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.226991 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-home\") pod \"edc515b2-4b5d-43d5-a561-59090a7ddd28\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " Jul 17 01:08:59.227060 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.227024 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4m2j\" (UniqueName: \"kubernetes.io/projected/edc515b2-4b5d-43d5-a561-59090a7ddd28-kube-api-access-p4m2j\") pod \"edc515b2-4b5d-43d5-a561-59090a7ddd28\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " Jul 17 01:08:59.227060 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.227044 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-model-cache\") pod \"edc515b2-4b5d-43d5-a561-59090a7ddd28\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " Jul 17 01:08:59.227178 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.227160 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc515b2-4b5d-43d5-a561-59090a7ddd28-tls-certs\") pod \"edc515b2-4b5d-43d5-a561-59090a7ddd28\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " Jul 17 01:08:59.227233 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.227222 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-tmp-dir\") pod \"edc515b2-4b5d-43d5-a561-59090a7ddd28\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " Jul 17 01:08:59.227281 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.227244 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-home" (OuterVolumeSpecName: "home") pod "edc515b2-4b5d-43d5-a561-59090a7ddd28" (UID: "edc515b2-4b5d-43d5-a561-59090a7ddd28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:08:59.227281 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.227253 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-dshm\") pod \"edc515b2-4b5d-43d5-a561-59090a7ddd28\" (UID: \"edc515b2-4b5d-43d5-a561-59090a7ddd28\") " Jul 17 01:08:59.227281 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.227267 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-model-cache" (OuterVolumeSpecName: "model-cache") pod "edc515b2-4b5d-43d5-a561-59090a7ddd28" (UID: "edc515b2-4b5d-43d5-a561-59090a7ddd28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:08:59.227488 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.227468 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edc515b2-4b5d-43d5-a561-59090a7ddd28" (UID: "edc515b2-4b5d-43d5-a561-59090a7ddd28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:08:59.228284 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.228052 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:08:59.228284 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.228076 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:08:59.228284 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.228091 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:08:59.229382 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.229352 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-dshm" (OuterVolumeSpecName: "dshm") pod "edc515b2-4b5d-43d5-a561-59090a7ddd28" (UID: "edc515b2-4b5d-43d5-a561-59090a7ddd28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:08:59.229484 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.229452 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc515b2-4b5d-43d5-a561-59090a7ddd28-kube-api-access-p4m2j" (OuterVolumeSpecName: "kube-api-access-p4m2j") pod "edc515b2-4b5d-43d5-a561-59090a7ddd28" (UID: "edc515b2-4b5d-43d5-a561-59090a7ddd28"). InnerVolumeSpecName "kube-api-access-p4m2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:08:59.229484 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.229474 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc515b2-4b5d-43d5-a561-59090a7ddd28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edc515b2-4b5d-43d5-a561-59090a7ddd28" (UID: "edc515b2-4b5d-43d5-a561-59090a7ddd28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:08:59.329245 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.329222 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4m2j\" (UniqueName: \"kubernetes.io/projected/edc515b2-4b5d-43d5-a561-59090a7ddd28-kube-api-access-p4m2j\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:08:59.329245 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.329242 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc515b2-4b5d-43d5-a561-59090a7ddd28-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:08:59.329364 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.329251 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc515b2-4b5d-43d5-a561-59090a7ddd28-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:08:59.537744 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.537721 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs"] Jul 17 01:08:59.539499 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.539480 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/auth-test-model-e1dd24-kserve-57579669f9-2ldxs"] Jul 17 01:08:59.736838 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.736796 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" podUID="2f74753a-1523-4f60-8950-e5e6dd9b3b49" containerName="main" containerID="cri-o://fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c" gracePeriod=45 Jul 17 01:08:59.977021 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:08:59.976994 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:09:00.091428 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.091386 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc515b2-4b5d-43d5-a561-59090a7ddd28" path="/var/lib/kubelet/pods/edc515b2-4b5d-43d5-a561-59090a7ddd28/volumes" Jul 17 01:09:00.136445 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136384 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6fp4\" (UniqueName: \"kubernetes.io/projected/2f74753a-1523-4f60-8950-e5e6dd9b3b49-kube-api-access-g6fp4\") pod \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " Jul 17 01:09:00.136445 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136416 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-model-cache\") pod \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " Jul 17 01:09:00.136445 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136442 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-dshm\") pod \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " Jul 17 01:09:00.136669 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136493 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-home\") pod \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " Jul 17 01:09:00.136669 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136575 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tls-certs\") pod \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " Jul 17 01:09:00.136669 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136597 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tmp-dir\") pod \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\" (UID: \"2f74753a-1523-4f60-8950-e5e6dd9b3b49\") " Jul 17 01:09:00.136798 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136680 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f74753a-1523-4f60-8950-e5e6dd9b3b49" (UID: "2f74753a-1523-4f60-8950-e5e6dd9b3b49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:09:00.136864 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136846 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:09:00.136864 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136829 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-home" (OuterVolumeSpecName: "home") pod "2f74753a-1523-4f60-8950-e5e6dd9b3b49" (UID: "2f74753a-1523-4f60-8950-e5e6dd9b3b49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:09:00.136990 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.136911 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f74753a-1523-4f60-8950-e5e6dd9b3b49" (UID: "2f74753a-1523-4f60-8950-e5e6dd9b3b49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:09:00.138683 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.138654 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-dshm" (OuterVolumeSpecName: "dshm") pod "2f74753a-1523-4f60-8950-e5e6dd9b3b49" (UID: "2f74753a-1523-4f60-8950-e5e6dd9b3b49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:09:00.138795 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.138737 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f74753a-1523-4f60-8950-e5e6dd9b3b49" (UID: "2f74753a-1523-4f60-8950-e5e6dd9b3b49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:09:00.138795 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.138764 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f74753a-1523-4f60-8950-e5e6dd9b3b49-kube-api-access-g6fp4" (OuterVolumeSpecName: "kube-api-access-g6fp4") pod "2f74753a-1523-4f60-8950-e5e6dd9b3b49" (UID: "2f74753a-1523-4f60-8950-e5e6dd9b3b49"). InnerVolumeSpecName "kube-api-access-g6fp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:09:00.222223 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.222202 2572 generic.go:356] "Generic (PLEG): container finished" podID="2f74753a-1523-4f60-8950-e5e6dd9b3b49" containerID="fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c" exitCode=0 Jul 17 01:09:00.222320 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.222240 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" event={"ID":"2f74753a-1523-4f60-8950-e5e6dd9b3b49","Type":"ContainerDied","Data":"fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c"} Jul 17 01:09:00.222320 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.222246 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" Jul 17 01:09:00.222320 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.222267 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g" event={"ID":"2f74753a-1523-4f60-8950-e5e6dd9b3b49","Type":"ContainerDied","Data":"79e75e7f7b933b9666f299a5100f3c923a96f4470ead48f472b4f47f0b4ad25c"} Jul 17 01:09:00.222320 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.222283 2572 scope.go:117] "RemoveContainer" containerID="fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c" Jul 17 01:09:00.233094 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.233076 2572 scope.go:117] "RemoveContainer" containerID="fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c" Jul 17 01:09:00.233343 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:09:00.233321 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c\": container with ID starting with fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c not found: ID does not exist" containerID="fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c" Jul 17 01:09:00.233416 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.233354 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c"} err="failed to get container status \"fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c\": rpc error: code = NotFound desc = could not find container \"fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c\": container with ID starting with fb0afb2e2a5ac6d98ea6ab683fd30b873012f35b6451eef05791267a7710ea8c not found: ID does not exist" Jul 17 01:09:00.237423 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.237401 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:09:00.237423 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.237423 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:09:00.237548 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.237436 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:09:00.237548 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.237448 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f74753a-1523-4f60-8950-e5e6dd9b3b49-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:09:00.237548 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.237459 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6fp4\" (UniqueName: \"kubernetes.io/projected/2f74753a-1523-4f60-8950-e5e6dd9b3b49-kube-api-access-g6fp4\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:09:00.244249 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.244227 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g"] Jul 17 01:09:00.247189 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:00.247169 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/auth-test-model-ac54e2-kserve-849778ff5-9vr9g"] Jul 17 01:09:02.087348 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:02.087307 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f74753a-1523-4f60-8950-e5e6dd9b3b49" path="/var/lib/kubelet/pods/2f74753a-1523-4f60-8950-e5e6dd9b3b49/volumes" Jul 17 01:09:03.409395 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:03.409361 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:09:03.409395 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:03.409403 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:09:03.421710 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:03.421683 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:09:04.252910 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:04.252862 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:09:06.101227 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:06.101190 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:09:11.298303 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.298265 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8"] Jul 17 01:09:11.298772 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.298670 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f74753a-1523-4f60-8950-e5e6dd9b3b49" containerName="main" Jul 17 01:09:11.298772 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.298685 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f74753a-1523-4f60-8950-e5e6dd9b3b49" containerName="main" Jul 17 01:09:11.298772 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.298712 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc515b2-4b5d-43d5-a561-59090a7ddd28" containerName="main" Jul 17 01:09:11.298772 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.298718 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc515b2-4b5d-43d5-a561-59090a7ddd28" containerName="main" Jul 17 01:09:11.298772 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.298727 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerName="revoke-keys" Jul 17 01:09:11.298772 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.298733 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc462517-6713-4988-bd31-fb243e8ad5eb" containerName="revoke-keys" Jul 17 01:09:11.299116 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.298789 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f74753a-1523-4f60-8950-e5e6dd9b3b49" containerName="main" Jul 17 01:09:11.299116 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.298798 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc515b2-4b5d-43d5-a561-59090a7ddd28" containerName="main" Jul 17 01:09:11.303384 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.303361 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.305872 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.305853 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"default-dockercfg-nhtws\"" Jul 17 01:09:11.306000 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.305853 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"sub-test-model-ac54e2-kserve-self-signed-certs\"" Jul 17 01:09:11.306000 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.305955 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"openshift-service-ca.crt\"" Jul 17 01:09:11.306113 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.306008 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"kube-root-ca.crt\"" Jul 17 01:09:11.310360 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.310342 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8"] Jul 17 01:09:11.330128 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.330103 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:09:11.337273 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.337245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-dshm\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.337365 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.337275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsgb4\" (UniqueName: \"kubernetes.io/projected/299e1820-5dbc-4cf7-9de5-6d7660c882c2-kube-api-access-nsgb4\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.337365 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.337301 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-home\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.337453 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.337374 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-model-cache\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.337453 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.337415 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tmp-dir\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.337453 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.337430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tls-certs\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.438523 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.438495 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tmp-dir\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.438523 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.438523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tls-certs\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.438682 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.438575 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-dshm\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.438682 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.438593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsgb4\" (UniqueName: \"kubernetes.io/projected/299e1820-5dbc-4cf7-9de5-6d7660c882c2-kube-api-access-nsgb4\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.438682 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.438616 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-home\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.438682 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.438646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-model-cache\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.438946 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.438923 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tmp-dir\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.439029 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.439009 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-model-cache\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.439094 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.439076 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-home\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.440725 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.440706 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-dshm\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.441008 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.440992 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tls-certs\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.446281 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.446260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsgb4\" (UniqueName: \"kubernetes.io/projected/299e1820-5dbc-4cf7-9de5-6d7660c882c2-kube-api-access-nsgb4\") pod \"sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.614800 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.614769 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:11.741573 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:11.741539 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8"] Jul 17 01:09:11.743646 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:09:11.743616 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod299e1820_5dbc_4cf7_9de5_6d7660c882c2.slice/crio-e8ebbe04e1b38a970053baed56342f64c5676921dd336e45fd6dd8dd2eaba364 WatchSource:0}: Error finding container e8ebbe04e1b38a970053baed56342f64c5676921dd336e45fd6dd8dd2eaba364: Status 404 returned error can't find the container with id e8ebbe04e1b38a970053baed56342f64c5676921dd336e45fd6dd8dd2eaba364 Jul 17 01:09:12.280511 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:12.280478 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" event={"ID":"299e1820-5dbc-4cf7-9de5-6d7660c882c2","Type":"ContainerStarted","Data":"7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658"} Jul 17 01:09:12.280511 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:12.280518 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" event={"ID":"299e1820-5dbc-4cf7-9de5-6d7660c882c2","Type":"ContainerStarted","Data":"e8ebbe04e1b38a970053baed56342f64c5676921dd336e45fd6dd8dd2eaba364"} Jul 17 01:09:12.296132 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:12.296088 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" podStartSLOduration=1.296073415 podStartE2EDuration="1.296073415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:09:12.294767058 +0000 UTC m=+3125.041276092" watchObservedRunningTime="2026-07-17 01:09:12.296073415 +0000 UTC m=+3125.042582448" Jul 17 01:09:19.002228 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:19.002190 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:09:21.615614 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:21.615573 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:21.616090 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:21.615629 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:21.628445 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:21.628422 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:22.340015 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:22.339984 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:09:45.986618 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:09:45.986581 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:10:06.596484 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:06.596447 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:10:12.479588 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:12.479555 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks"] Jul 17 01:10:12.503298 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:12.503264 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:10:13.386602 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:13.386571 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8"] Jul 17 01:10:13.402049 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:13.402011 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:10:26.386082 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:26.386047 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:10:27.500509 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.500457 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" podUID="38781528-cb08-4082-a5dc-6cf8a1e974da" containerName="main" containerID="cri-o://b0060c81c423a6001da00666310860835edd85e88239af65bc5cdfe7ec0cfd3b" gracePeriod=45 Jul 17 01:10:27.651548 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.651518 2572 generic.go:356] "Generic (PLEG): container finished" podID="38781528-cb08-4082-a5dc-6cf8a1e974da" containerID="b0060c81c423a6001da00666310860835edd85e88239af65bc5cdfe7ec0cfd3b" exitCode=0 Jul 17 01:10:27.651724 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.651593 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" event={"ID":"38781528-cb08-4082-a5dc-6cf8a1e974da","Type":"ContainerDied","Data":"b0060c81c423a6001da00666310860835edd85e88239af65bc5cdfe7ec0cfd3b"} Jul 17 01:10:27.744827 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.744803 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:10:27.816743 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.816653 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-tmp-dir\") pod \"38781528-cb08-4082-a5dc-6cf8a1e974da\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " Jul 17 01:10:27.816743 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.816711 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-dshm\") pod \"38781528-cb08-4082-a5dc-6cf8a1e974da\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " Jul 17 01:10:27.817017 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.816780 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgxqv\" (UniqueName: \"kubernetes.io/projected/38781528-cb08-4082-a5dc-6cf8a1e974da-kube-api-access-dgxqv\") pod \"38781528-cb08-4082-a5dc-6cf8a1e974da\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " Jul 17 01:10:27.817017 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.816835 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-home\") pod \"38781528-cb08-4082-a5dc-6cf8a1e974da\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " Jul 17 01:10:27.817017 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.816943 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-model-cache\") pod \"38781528-cb08-4082-a5dc-6cf8a1e974da\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " Jul 17 01:10:27.817017 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.816969 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38781528-cb08-4082-a5dc-6cf8a1e974da" (UID: "38781528-cb08-4082-a5dc-6cf8a1e974da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:10:27.817239 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.817050 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38781528-cb08-4082-a5dc-6cf8a1e974da-tls-certs\") pod \"38781528-cb08-4082-a5dc-6cf8a1e974da\" (UID: \"38781528-cb08-4082-a5dc-6cf8a1e974da\") " Jul 17 01:10:27.817239 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.817051 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-home" (OuterVolumeSpecName: "home") pod "38781528-cb08-4082-a5dc-6cf8a1e974da" (UID: "38781528-cb08-4082-a5dc-6cf8a1e974da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:10:27.817239 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.817177 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-model-cache" (OuterVolumeSpecName: "model-cache") pod "38781528-cb08-4082-a5dc-6cf8a1e974da" (UID: "38781528-cb08-4082-a5dc-6cf8a1e974da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:10:27.817386 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.817353 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:27.817386 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.817369 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:27.817386 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.817382 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:27.818855 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.818825 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-dshm" (OuterVolumeSpecName: "dshm") pod "38781528-cb08-4082-a5dc-6cf8a1e974da" (UID: "38781528-cb08-4082-a5dc-6cf8a1e974da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:10:27.818855 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.818839 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38781528-cb08-4082-a5dc-6cf8a1e974da-kube-api-access-dgxqv" (OuterVolumeSpecName: "kube-api-access-dgxqv") pod "38781528-cb08-4082-a5dc-6cf8a1e974da" (UID: "38781528-cb08-4082-a5dc-6cf8a1e974da"). InnerVolumeSpecName "kube-api-access-dgxqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:10:27.819030 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.818945 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38781528-cb08-4082-a5dc-6cf8a1e974da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38781528-cb08-4082-a5dc-6cf8a1e974da" (UID: "38781528-cb08-4082-a5dc-6cf8a1e974da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:10:27.918281 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.918251 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38781528-cb08-4082-a5dc-6cf8a1e974da-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:27.918281 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.918281 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgxqv\" (UniqueName: \"kubernetes.io/projected/38781528-cb08-4082-a5dc-6cf8a1e974da-kube-api-access-dgxqv\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:27.918475 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:27.918292 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38781528-cb08-4082-a5dc-6cf8a1e974da-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:28.406864 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.406756 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" podUID="299e1820-5dbc-4cf7-9de5-6d7660c882c2" containerName="main" containerID="cri-o://7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658" gracePeriod=45 Jul 17 01:10:28.655584 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.655550 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:10:28.656964 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.656912 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" Jul 17 01:10:28.656964 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.656913 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks" event={"ID":"38781528-cb08-4082-a5dc-6cf8a1e974da","Type":"ContainerDied","Data":"5096762431ca3de23f13141cfbfc2679a57539fe68d458a54643980eb41d2ade"} Jul 17 01:10:28.656964 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.656962 2572 scope.go:117] "RemoveContainer" containerID="b0060c81c423a6001da00666310860835edd85e88239af65bc5cdfe7ec0cfd3b" Jul 17 01:10:28.658192 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.658163 2572 generic.go:356] "Generic (PLEG): container finished" podID="299e1820-5dbc-4cf7-9de5-6d7660c882c2" containerID="7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658" exitCode=0 Jul 17 01:10:28.658314 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.658194 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" Jul 17 01:10:28.658314 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.658245 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" event={"ID":"299e1820-5dbc-4cf7-9de5-6d7660c882c2","Type":"ContainerDied","Data":"7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658"} Jul 17 01:10:28.658314 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.658299 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8" event={"ID":"299e1820-5dbc-4cf7-9de5-6d7660c882c2","Type":"ContainerDied","Data":"e8ebbe04e1b38a970053baed56342f64c5676921dd336e45fd6dd8dd2eaba364"} Jul 17 01:10:28.665260 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.665242 2572 scope.go:117] "RemoveContainer" containerID="7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658" Jul 17 01:10:28.672148 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.672118 2572 scope.go:117] "RemoveContainer" containerID="7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658" Jul 17 01:10:28.676208 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:10:28.676134 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658\": container with ID starting with 7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658 not found: ID does not exist" containerID="7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658" Jul 17 01:10:28.676208 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.676171 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658"} err="failed to get container status \"7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658\": rpc error: code = NotFound desc = could not find container \"7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658\": container with ID starting with 7ca4f2e115610d433e2417b807f1c6e578eb8f4f9ec13245c69120d873174658 not found: ID does not exist" Jul 17 01:10:28.687997 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.687973 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks"] Jul 17 01:10:28.693916 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.693874 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/sub-test-model-e1dd24-kserve-8c4bbfc5f-pkkks"] Jul 17 01:10:28.726789 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.726766 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-dshm\") pod \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " Jul 17 01:10:28.726875 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.726803 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-model-cache\") pod \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " Jul 17 01:10:28.726875 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.726845 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsgb4\" (UniqueName: \"kubernetes.io/projected/299e1820-5dbc-4cf7-9de5-6d7660c882c2-kube-api-access-nsgb4\") pod \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " Jul 17 01:10:28.726991 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.726883 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tmp-dir\") pod \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " Jul 17 01:10:28.726991 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.726951 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tls-certs\") pod \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " Jul 17 01:10:28.727367 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.726994 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-home\") pod \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\" (UID: \"299e1820-5dbc-4cf7-9de5-6d7660c882c2\") " Jul 17 01:10:28.727367 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.727094 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "299e1820-5dbc-4cf7-9de5-6d7660c882c2" (UID: "299e1820-5dbc-4cf7-9de5-6d7660c882c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:10:28.727367 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.727182 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "299e1820-5dbc-4cf7-9de5-6d7660c882c2" (UID: "299e1820-5dbc-4cf7-9de5-6d7660c882c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:10:28.727367 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.727249 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-home" (OuterVolumeSpecName: "home") pod "299e1820-5dbc-4cf7-9de5-6d7660c882c2" (UID: "299e1820-5dbc-4cf7-9de5-6d7660c882c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:10:28.727367 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.727357 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:28.727570 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.727376 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:28.727570 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.727388 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:28.729076 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.729044 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-dshm" (OuterVolumeSpecName: "dshm") pod "299e1820-5dbc-4cf7-9de5-6d7660c882c2" (UID: "299e1820-5dbc-4cf7-9de5-6d7660c882c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:10:28.729165 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.729118 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "299e1820-5dbc-4cf7-9de5-6d7660c882c2" (UID: "299e1820-5dbc-4cf7-9de5-6d7660c882c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:10:28.729229 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.729176 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299e1820-5dbc-4cf7-9de5-6d7660c882c2-kube-api-access-nsgb4" (OuterVolumeSpecName: "kube-api-access-nsgb4") pod "299e1820-5dbc-4cf7-9de5-6d7660c882c2" (UID: "299e1820-5dbc-4cf7-9de5-6d7660c882c2"). InnerVolumeSpecName "kube-api-access-nsgb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:10:28.828308 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.828284 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299e1820-5dbc-4cf7-9de5-6d7660c882c2-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:28.828417 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.828314 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsgb4\" (UniqueName: \"kubernetes.io/projected/299e1820-5dbc-4cf7-9de5-6d7660c882c2-kube-api-access-nsgb4\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:28.828417 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.828325 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299e1820-5dbc-4cf7-9de5-6d7660c882c2-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:28.830954 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.830931 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:10:28.934556 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.934481 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:10:28.984036 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.983625 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8"] Jul 17 01:10:28.984036 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:28.983672 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/sub-test-model-ac54e2-kserve-5d4fbfb4cf-cnvb8"] Jul 17 01:10:30.087556 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:30.087522 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="299e1820-5dbc-4cf7-9de5-6d7660c882c2" path="/var/lib/kubelet/pods/299e1820-5dbc-4cf7-9de5-6d7660c882c2/volumes" Jul 17 01:10:30.087927 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:30.087882 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38781528-cb08-4082-a5dc-6cf8a1e974da" path="/var/lib/kubelet/pods/38781528-cb08-4082-a5dc-6cf8a1e974da/volumes" Jul 17 01:10:31.103719 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.103690 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp"] Jul 17 01:10:31.104086 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.104042 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="299e1820-5dbc-4cf7-9de5-6d7660c882c2" containerName="main" Jul 17 01:10:31.104086 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.104053 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="299e1820-5dbc-4cf7-9de5-6d7660c882c2" containerName="main" Jul 17 01:10:31.104086 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.104068 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38781528-cb08-4082-a5dc-6cf8a1e974da" containerName="main" Jul 17 01:10:31.104086 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.104074 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="38781528-cb08-4082-a5dc-6cf8a1e974da" containerName="main" Jul 17 01:10:31.104234 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.104141 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="38781528-cb08-4082-a5dc-6cf8a1e974da" containerName="main" Jul 17 01:10:31.104234 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.104148 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="299e1820-5dbc-4cf7-9de5-6d7660c882c2" containerName="main" Jul 17 01:10:31.108727 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.108706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.111119 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.111097 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e1dd24\"/\"openshift-service-ca.crt\"" Jul 17 01:10:31.112019 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.112002 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e1dd24\"/\"kube-root-ca.crt\"" Jul 17 01:10:31.112135 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.112072 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e1dd24\"/\"default-dockercfg-64w99\"" Jul 17 01:10:31.112135 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.112088 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e1dd24\"/\"rate-test-model-e1dd24-kserve-self-signed-certs\"" Jul 17 01:10:31.115177 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.115154 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp"] Jul 17 01:10:31.145927 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.145881 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-dshm\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.146031 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.145944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tls-certs\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.146074 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.146043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8zj\" (UniqueName: \"kubernetes.io/projected/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-kube-api-access-bs8zj\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.146120 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.146070 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-home\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.146120 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.146088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tmp-dir\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.146238 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.146219 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-model-cache\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.247639 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.247606 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8zj\" (UniqueName: \"kubernetes.io/projected/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-kube-api-access-bs8zj\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.247639 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.247642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-home\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.247912 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.247661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tmp-dir\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.247912 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.247716 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-model-cache\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.247912 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.247768 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-dshm\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.247912 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.247801 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tls-certs\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.248130 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.248066 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-home\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.248130 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.248111 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tmp-dir\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.248293 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.248180 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-model-cache\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.250040 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.250017 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-dshm\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.250304 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.250287 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tls-certs\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.255153 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.255126 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8zj\" (UniqueName: \"kubernetes.io/projected/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-kube-api-access-bs8zj\") pod \"rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.419014 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.418932 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:31.545353 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.545287 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp"] Jul 17 01:10:31.547545 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:10:31.547523 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0bddca1_6d0f_46a1_9bfa_df0b8594b439.slice/crio-4d4cf74a09eb00f9e88bf34d82ed56b9fc72f2de66bba443a0bacb1c9eac7402 WatchSource:0}: Error finding container 4d4cf74a09eb00f9e88bf34d82ed56b9fc72f2de66bba443a0bacb1c9eac7402: Status 404 returned error can't find the container with id 4d4cf74a09eb00f9e88bf34d82ed56b9fc72f2de66bba443a0bacb1c9eac7402 Jul 17 01:10:31.676053 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.675968 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" event={"ID":"b0bddca1-6d0f-46a1-9bfa-df0b8594b439","Type":"ContainerStarted","Data":"134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2"} Jul 17 01:10:31.676053 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.676011 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" event={"ID":"b0bddca1-6d0f-46a1-9bfa-df0b8594b439","Type":"ContainerStarted","Data":"4d4cf74a09eb00f9e88bf34d82ed56b9fc72f2de66bba443a0bacb1c9eac7402"} Jul 17 01:10:31.691103 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:31.691049 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" podStartSLOduration=0.691031479 podStartE2EDuration="691.031479ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:10:31.690621111 +0000 UTC m=+3204.437130145" watchObservedRunningTime="2026-07-17 01:10:31.691031479 +0000 UTC m=+3204.437540514" Jul 17 01:10:33.546848 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:33.546814 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:10:35.896674 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:35.896638 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:10:36.684011 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:36.683976 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn"] Jul 17 01:10:36.687493 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:36.687477 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" Jul 17 01:10:36.689842 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:36.689824 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\"" Jul 17 01:10:36.689968 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:36.689821 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:10:36.693645 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:36.693619 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn"] Jul 17 01:10:36.797505 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:36.797477 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebafa24e-ca55-43e8-9823-1c70a4d11806-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-hs6cn\" (UID: \"ebafa24e-ca55-43e8-9823-1c70a4d11806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" Jul 17 01:10:36.898800 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:36.898774 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebafa24e-ca55-43e8-9823-1c70a4d11806-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-hs6cn\" (UID: \"ebafa24e-ca55-43e8-9823-1c70a4d11806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" Jul 17 01:10:36.899358 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:36.899338 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebafa24e-ca55-43e8-9823-1c70a4d11806-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-732542a6-hs6cn\" (UID: \"ebafa24e-ca55-43e8-9823-1c70a4d11806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" Jul 17 01:10:37.000161 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:37.000101 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" Jul 17 01:10:37.126848 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:37.126820 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn"] Jul 17 01:10:37.128750 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:10:37.128725 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebafa24e_ca55_43e8_9823_1c70a4d11806.slice/crio-4b9ef0426047f0006d352828616dc8517515864cbb3fb441a4e05b9cc3250c50 WatchSource:0}: Error finding container 4b9ef0426047f0006d352828616dc8517515864cbb3fb441a4e05b9cc3250c50: Status 404 returned error can't find the container with id 4b9ef0426047f0006d352828616dc8517515864cbb3fb441a4e05b9cc3250c50 Jul 17 01:10:37.708906 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:37.708868 2572 generic.go:356] "Generic (PLEG): container finished" podID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerID="cfcbfb5413d1468699b9701df249a97a40f5d92565eb7002a1837189397728d1" exitCode=6 Jul 17 01:10:37.709052 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:37.708915 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" event={"ID":"ebafa24e-ca55-43e8-9823-1c70a4d11806","Type":"ContainerDied","Data":"cfcbfb5413d1468699b9701df249a97a40f5d92565eb7002a1837189397728d1"} Jul 17 01:10:37.709052 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:37.708951 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" event={"ID":"ebafa24e-ca55-43e8-9823-1c70a4d11806","Type":"ContainerStarted","Data":"4b9ef0426047f0006d352828616dc8517515864cbb3fb441a4e05b9cc3250c50"} Jul 17 01:10:37.709131 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:37.709084 2572 scope.go:117] "RemoveContainer" containerID="cfcbfb5413d1468699b9701df249a97a40f5d92565eb7002a1837189397728d1" Jul 17 01:10:38.715818 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:38.715785 2572 generic.go:356] "Generic (PLEG): container finished" podID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerID="f2701daf96f36cb0c5cb7e5d55c2fa4c8b6b7bacdcf00f410abef6843f4f990e" exitCode=6 Jul 17 01:10:38.716297 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:38.715831 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" event={"ID":"ebafa24e-ca55-43e8-9823-1c70a4d11806","Type":"ContainerDied","Data":"f2701daf96f36cb0c5cb7e5d55c2fa4c8b6b7bacdcf00f410abef6843f4f990e"} Jul 17 01:10:38.716297 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:38.715874 2572 scope.go:117] "RemoveContainer" containerID="cfcbfb5413d1468699b9701df249a97a40f5d92565eb7002a1837189397728d1" Jul 17 01:10:38.716297 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:38.716113 2572 scope.go:117] "RemoveContainer" containerID="f2701daf96f36cb0c5cb7e5d55c2fa4c8b6b7bacdcf00f410abef6843f4f990e" Jul 17 01:10:38.716473 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:10:38.716337 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-hs6cn_odh-ai-gateway-infra(ebafa24e-ca55-43e8-9823-1c70a4d11806)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" Jul 17 01:10:39.722574 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:39.722546 2572 scope.go:117] "RemoveContainer" containerID="f2701daf96f36cb0c5cb7e5d55c2fa4c8b6b7bacdcf00f410abef6843f4f990e" Jul 17 01:10:39.722994 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:10:39.722710 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-hs6cn_odh-ai-gateway-infra(ebafa24e-ca55-43e8-9823-1c70a4d11806)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" Jul 17 01:10:41.419862 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:41.419829 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:41.420341 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:41.419873 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:41.432296 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:41.432274 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:41.450760 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:41.450732 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:10:41.742756 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:41.742674 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:10:47.078492 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.078456 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8"] Jul 17 01:10:47.083314 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.083291 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.086649 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.086627 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"openshift-service-ca.crt\"" Jul 17 01:10:47.086649 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.086646 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"default-dockercfg-nhtws\"" Jul 17 01:10:47.086826 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.086627 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"rate-test-model-ac54e2-kserve-self-signed-certs\"" Jul 17 01:10:47.086826 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.086636 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac54e2\"/\"kube-root-ca.crt\"" Jul 17 01:10:47.090404 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.090380 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8"] Jul 17 01:10:47.107183 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.107153 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:10:47.184739 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.184713 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-tmp-dir\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.184849 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.184752 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50c22b-c87e-4be3-96b9-155f3491663b-tls-certs\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.184849 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.184789 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-dshm\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.184849 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.184809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wsj7\" (UniqueName: \"kubernetes.io/projected/8a50c22b-c87e-4be3-96b9-155f3491663b-kube-api-access-4wsj7\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.184849 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.184829 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-home\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.184849 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.184843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-model-cache\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.285716 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.285678 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-dshm\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.285716 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.285718 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsj7\" (UniqueName: \"kubernetes.io/projected/8a50c22b-c87e-4be3-96b9-155f3491663b-kube-api-access-4wsj7\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.286013 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.285849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-home\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.286013 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.285926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-model-cache\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.286117 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.286026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-tmp-dir\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.286117 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.286070 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50c22b-c87e-4be3-96b9-155f3491663b-tls-certs\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.286275 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.286247 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-home\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.286427 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.286278 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-model-cache\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.286427 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.286410 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-tmp-dir\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.288002 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.287983 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-dshm\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.288380 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.288362 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50c22b-c87e-4be3-96b9-155f3491663b-tls-certs\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.292988 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.292970 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsj7\" (UniqueName: \"kubernetes.io/projected/8a50c22b-c87e-4be3-96b9-155f3491663b-kube-api-access-4wsj7\") pod \"rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.393321 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.393283 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:47.528725 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.528701 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8"] Jul 17 01:10:47.531221 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:10:47.531192 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a50c22b_c87e_4be3_96b9_155f3491663b.slice/crio-e4c33ddc689470f37e8cfaa32ed75f972da6624479fb4efb36c0e8cd2eee799f WatchSource:0}: Error finding container e4c33ddc689470f37e8cfaa32ed75f972da6624479fb4efb36c0e8cd2eee799f: Status 404 returned error can't find the container with id e4c33ddc689470f37e8cfaa32ed75f972da6624479fb4efb36c0e8cd2eee799f Jul 17 01:10:47.760409 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.760321 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" event={"ID":"8a50c22b-c87e-4be3-96b9-155f3491663b","Type":"ContainerStarted","Data":"35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c"} Jul 17 01:10:47.760409 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.760363 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" event={"ID":"8a50c22b-c87e-4be3-96b9-155f3491663b","Type":"ContainerStarted","Data":"e4c33ddc689470f37e8cfaa32ed75f972da6624479fb4efb36c0e8cd2eee799f"} Jul 17 01:10:47.775694 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:47.775649 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" podStartSLOduration=0.775635033 podStartE2EDuration="775.635033ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:10:47.775121169 +0000 UTC m=+3220.521630220" watchObservedRunningTime="2026-07-17 01:10:47.775635033 +0000 UTC m=+3220.522144066" Jul 17 01:10:54.082481 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:54.082453 2572 scope.go:117] "RemoveContainer" containerID="f2701daf96f36cb0c5cb7e5d55c2fa4c8b6b7bacdcf00f410abef6843f4f990e" Jul 17 01:10:54.603009 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:54.602979 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:10:54.794548 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:54.794517 2572 generic.go:356] "Generic (PLEG): container finished" podID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerID="cc736b10bab0a76092ddac933f33448a2f8e9a224fda80a02ca9da80ae1e0518" exitCode=6 Jul 17 01:10:54.794691 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:54.794559 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" event={"ID":"ebafa24e-ca55-43e8-9823-1c70a4d11806","Type":"ContainerDied","Data":"cc736b10bab0a76092ddac933f33448a2f8e9a224fda80a02ca9da80ae1e0518"} Jul 17 01:10:54.794691 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:54.794606 2572 scope.go:117] "RemoveContainer" containerID="f2701daf96f36cb0c5cb7e5d55c2fa4c8b6b7bacdcf00f410abef6843f4f990e" Jul 17 01:10:54.794856 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:54.794839 2572 scope.go:117] "RemoveContainer" containerID="cc736b10bab0a76092ddac933f33448a2f8e9a224fda80a02ca9da80ae1e0518" Jul 17 01:10:54.795098 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:10:54.795079 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-732542a6-hs6cn_odh-ai-gateway-infra(ebafa24e-ca55-43e8-9823-1c70a4d11806)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" Jul 17 01:10:55.820135 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:55.820109 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn"] Jul 17 01:10:55.952828 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:55.952804 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" Jul 17 01:10:56.063182 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:56.063148 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebafa24e-ca55-43e8-9823-1c70a4d11806-maas-api-service-ca\") pod \"ebafa24e-ca55-43e8-9823-1c70a4d11806\" (UID: \"ebafa24e-ca55-43e8-9823-1c70a4d11806\") " Jul 17 01:10:56.063493 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:56.063471 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebafa24e-ca55-43e8-9823-1c70a4d11806-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebafa24e-ca55-43e8-9823-1c70a4d11806" (UID: "ebafa24e-ca55-43e8-9823-1c70a4d11806"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:10:56.164590 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:56.164544 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebafa24e-ca55-43e8-9823-1c70a4d11806-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:10:56.807272 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:56.807234 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" Jul 17 01:10:56.807450 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:56.807238 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn" event={"ID":"ebafa24e-ca55-43e8-9823-1c70a4d11806","Type":"ContainerDied","Data":"4b9ef0426047f0006d352828616dc8517515864cbb3fb441a4e05b9cc3250c50"} Jul 17 01:10:56.807450 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:56.807390 2572 scope.go:117] "RemoveContainer" containerID="cc736b10bab0a76092ddac933f33448a2f8e9a224fda80a02ca9da80ae1e0518" Jul 17 01:10:56.823094 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:56.823066 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn"] Jul 17 01:10:56.824857 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:56.824836 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-732542a6-hs6cn"] Jul 17 01:10:57.393581 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:57.393542 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:57.393726 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:57.393594 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:57.406073 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:57.406046 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:57.823836 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:57.823760 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:10:58.087522 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:10:58.087439 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" path="/var/lib/kubelet/pods/ebafa24e-ca55-43e8-9823-1c70a4d11806/volumes" Jul 17 01:11:09.333643 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:09.333598 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:11:09.396588 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:09.396552 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:11:09.397961 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:09.397935 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:11:16.685899 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:16.685854 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:11:16.786665 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:16.786628 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:11:29.193902 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:29.193856 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:11:36.353488 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:36.353453 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp"] Jul 17 01:11:36.365676 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:36.365625 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:11:37.303576 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:37.303538 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8"] Jul 17 01:11:37.320803 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:37.320756 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:11:38.236635 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.236604 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7469-139661443684528"] Jul 17 01:11:38.237030 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.236991 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerName="revoke-keys" Jul 17 01:11:38.237030 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.237003 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerName="revoke-keys" Jul 17 01:11:38.237030 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.237014 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerName="revoke-keys" Jul 17 01:11:38.237030 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.237020 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerName="revoke-keys" Jul 17 01:11:38.237166 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.237037 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerName="revoke-keys" Jul 17 01:11:38.237166 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.237042 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerName="revoke-keys" Jul 17 01:11:38.237166 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.237099 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerName="revoke-keys" Jul 17 01:11:38.237166 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.237108 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerName="revoke-keys" Jul 17 01:11:38.237166 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.237117 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebafa24e-ca55-43e8-9823-1c70a4d11806" containerName="revoke-keys" Jul 17 01:11:38.240364 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.240347 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443684528" Jul 17 01:11:38.242572 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.242548 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czq5d\"" Jul 17 01:11:38.245443 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.245421 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443684528"] Jul 17 01:11:38.343542 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.343510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhlj\" (UniqueName: \"kubernetes.io/projected/510478a4-20c5-4368-91cc-76f46231f138-kube-api-access-cwhlj\") pod \"test-curl-7469-139661443684528\" (UID: \"510478a4-20c5-4368-91cc-76f46231f138\") " pod="opendatahub/test-curl-7469-139661443684528" Jul 17 01:11:38.444485 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.444451 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhlj\" (UniqueName: \"kubernetes.io/projected/510478a4-20c5-4368-91cc-76f46231f138-kube-api-access-cwhlj\") pod \"test-curl-7469-139661443684528\" (UID: \"510478a4-20c5-4368-91cc-76f46231f138\") " pod="opendatahub/test-curl-7469-139661443684528" Jul 17 01:11:38.452936 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.452905 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhlj\" (UniqueName: \"kubernetes.io/projected/510478a4-20c5-4368-91cc-76f46231f138-kube-api-access-cwhlj\") pod \"test-curl-7469-139661443684528\" (UID: \"510478a4-20c5-4368-91cc-76f46231f138\") " pod="opendatahub/test-curl-7469-139661443684528" Jul 17 01:11:38.551134 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.551045 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443684528" Jul 17 01:11:38.682094 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.681912 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443684528"] Jul 17 01:11:38.684277 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:11:38.684253 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod510478a4_20c5_4368_91cc_76f46231f138.slice/crio-d91f4e07bbd0ed87c85b117563322dd7db64d2ea43fbc3a67174452344c72e24 WatchSource:0}: Error finding container d91f4e07bbd0ed87c85b117563322dd7db64d2ea43fbc3a67174452344c72e24: Status 404 returned error can't find the container with id d91f4e07bbd0ed87c85b117563322dd7db64d2ea43fbc3a67174452344c72e24 Jul 17 01:11:38.685664 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:38.685644 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 01:11:39.005386 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:39.005353 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443684528" event={"ID":"510478a4-20c5-4368-91cc-76f46231f138","Type":"ContainerStarted","Data":"d91f4e07bbd0ed87c85b117563322dd7db64d2ea43fbc3a67174452344c72e24"} Jul 17 01:11:40.011056 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:40.011021 2572 generic.go:356] "Generic (PLEG): container finished" podID="510478a4-20c5-4368-91cc-76f46231f138" containerID="83c771449acdfe1afd2ada82085a071b1a6c0eae3a893c2c7cbb9d9d1f4c9533" exitCode=0 Jul 17 01:11:40.011446 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:40.011109 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443684528" event={"ID":"510478a4-20c5-4368-91cc-76f46231f138","Type":"ContainerDied","Data":"83c771449acdfe1afd2ada82085a071b1a6c0eae3a893c2c7cbb9d9d1f4c9533"} Jul 17 01:11:41.143184 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.143158 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443684528" Jul 17 01:11:41.162167 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.162128 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7469-139661443684528_510478a4-20c5-4368-91cc-76f46231f138/test-curl-7469-139661443684528/0.log" Jul 17 01:11:41.184027 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.183997 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7469-139661443684528"] Jul 17 01:11:41.187363 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.187341 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7469-139661443684528"] Jul 17 01:11:41.268570 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.268485 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwhlj\" (UniqueName: \"kubernetes.io/projected/510478a4-20c5-4368-91cc-76f46231f138-kube-api-access-cwhlj\") pod \"510478a4-20c5-4368-91cc-76f46231f138\" (UID: \"510478a4-20c5-4368-91cc-76f46231f138\") " Jul 17 01:11:41.270730 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.270702 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510478a4-20c5-4368-91cc-76f46231f138-kube-api-access-cwhlj" (OuterVolumeSpecName: "kube-api-access-cwhlj") pod "510478a4-20c5-4368-91cc-76f46231f138" (UID: "510478a4-20c5-4368-91cc-76f46231f138"). InnerVolumeSpecName "kube-api-access-cwhlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:11:41.303483 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.303451 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7469-139661443684656"] Jul 17 01:11:41.303838 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.303825 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510478a4-20c5-4368-91cc-76f46231f138" containerName="test-curl-7469-139661443684528" Jul 17 01:11:41.303899 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.303839 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="510478a4-20c5-4368-91cc-76f46231f138" containerName="test-curl-7469-139661443684528" Jul 17 01:11:41.303939 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.303918 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="510478a4-20c5-4368-91cc-76f46231f138" containerName="test-curl-7469-139661443684528" Jul 17 01:11:41.307079 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.307061 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443684656" Jul 17 01:11:41.313278 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.313253 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443684656"] Jul 17 01:11:41.369504 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.369474 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhlj\" (UniqueName: \"kubernetes.io/projected/510478a4-20c5-4368-91cc-76f46231f138-kube-api-access-cwhlj\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:41.470876 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.470837 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbf7\" (UniqueName: \"kubernetes.io/projected/6e01bda7-1fb3-43e0-9197-f5627fd71fd8-kube-api-access-tsbf7\") pod \"test-curl-7469-139661443684656\" (UID: \"6e01bda7-1fb3-43e0-9197-f5627fd71fd8\") " pod="opendatahub/test-curl-7469-139661443684656" Jul 17 01:11:41.571677 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.571588 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbf7\" (UniqueName: \"kubernetes.io/projected/6e01bda7-1fb3-43e0-9197-f5627fd71fd8-kube-api-access-tsbf7\") pod \"test-curl-7469-139661443684656\" (UID: \"6e01bda7-1fb3-43e0-9197-f5627fd71fd8\") " pod="opendatahub/test-curl-7469-139661443684656" Jul 17 01:11:41.579485 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.579460 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbf7\" (UniqueName: \"kubernetes.io/projected/6e01bda7-1fb3-43e0-9197-f5627fd71fd8-kube-api-access-tsbf7\") pod \"test-curl-7469-139661443684656\" (UID: \"6e01bda7-1fb3-43e0-9197-f5627fd71fd8\") " pod="opendatahub/test-curl-7469-139661443684656" Jul 17 01:11:41.617679 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.617634 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443684656" Jul 17 01:11:41.745857 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:41.745835 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443684656"] Jul 17 01:11:41.747616 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:11:41.747585 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e01bda7_1fb3_43e0_9197_f5627fd71fd8.slice/crio-5c7c8c9ee571fa18c267659ea966540d51b58d7b799f2b351e02c52bf65edf67 WatchSource:0}: Error finding container 5c7c8c9ee571fa18c267659ea966540d51b58d7b799f2b351e02c52bf65edf67: Status 404 returned error can't find the container with id 5c7c8c9ee571fa18c267659ea966540d51b58d7b799f2b351e02c52bf65edf67 Jul 17 01:11:42.022819 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:42.022790 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443684528" Jul 17 01:11:42.023031 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:42.022828 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d91f4e07bbd0ed87c85b117563322dd7db64d2ea43fbc3a67174452344c72e24" Jul 17 01:11:42.024130 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:42.024094 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443684656" event={"ID":"6e01bda7-1fb3-43e0-9197-f5627fd71fd8","Type":"ContainerStarted","Data":"5c7c8c9ee571fa18c267659ea966540d51b58d7b799f2b351e02c52bf65edf67"} Jul 17 01:11:42.033497 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:11:42.033470 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7469-139661443684528\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="510478a4-20c5-4368-91cc-76f46231f138" pod="opendatahub/test-curl-7469-139661443684528" Jul 17 01:11:42.091685 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:42.091655 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="510478a4-20c5-4368-91cc-76f46231f138" path="/var/lib/kubelet/pods/510478a4-20c5-4368-91cc-76f46231f138/volumes" Jul 17 01:11:43.030382 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:43.030352 2572 generic.go:356] "Generic (PLEG): container finished" podID="6e01bda7-1fb3-43e0-9197-f5627fd71fd8" containerID="21f2a1814b554ddad2c1e23fb2c5fbcb80e21226b26b459e70fbeb0ce0b8e022" exitCode=0 Jul 17 01:11:43.030768 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:43.030396 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443684656" event={"ID":"6e01bda7-1fb3-43e0-9197-f5627fd71fd8","Type":"ContainerDied","Data":"21f2a1814b554ddad2c1e23fb2c5fbcb80e21226b26b459e70fbeb0ce0b8e022"} Jul 17 01:11:44.163349 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:44.163322 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443684656" Jul 17 01:11:44.180950 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:44.180918 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7469-139661443684656_6e01bda7-1fb3-43e0-9197-f5627fd71fd8/test-curl-7469-139661443684656/0.log" Jul 17 01:11:44.199513 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:44.199477 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7469-139661443684656"] Jul 17 01:11:44.202719 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:44.202690 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7469-139661443684656"] Jul 17 01:11:44.296733 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:44.296638 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsbf7\" (UniqueName: \"kubernetes.io/projected/6e01bda7-1fb3-43e0-9197-f5627fd71fd8-kube-api-access-tsbf7\") pod \"6e01bda7-1fb3-43e0-9197-f5627fd71fd8\" (UID: \"6e01bda7-1fb3-43e0-9197-f5627fd71fd8\") " Jul 17 01:11:44.298688 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:44.298666 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e01bda7-1fb3-43e0-9197-f5627fd71fd8-kube-api-access-tsbf7" (OuterVolumeSpecName: "kube-api-access-tsbf7") pod "6e01bda7-1fb3-43e0-9197-f5627fd71fd8" (UID: "6e01bda7-1fb3-43e0-9197-f5627fd71fd8"). InnerVolumeSpecName "kube-api-access-tsbf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:11:44.398126 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:44.398088 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsbf7\" (UniqueName: \"kubernetes.io/projected/6e01bda7-1fb3-43e0-9197-f5627fd71fd8-kube-api-access-tsbf7\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:44.894280 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:44.894243 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:11:45.041166 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:45.041134 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443684656" Jul 17 01:11:45.041335 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:45.041135 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c7c8c9ee571fa18c267659ea966540d51b58d7b799f2b351e02c52bf65edf67" Jul 17 01:11:45.050714 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:11:45.050689 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7469-139661443684656\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="6e01bda7-1fb3-43e0-9197-f5627fd71fd8" pod="opendatahub/test-curl-7469-139661443684656" Jul 17 01:11:46.087884 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:46.087850 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e01bda7-1fb3-43e0-9197-f5627fd71fd8" path="/var/lib/kubelet/pods/6e01bda7-1fb3-43e0-9197-f5627fd71fd8/volumes" Jul 17 01:11:47.816389 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.816354 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7469-139661443547760"] Jul 17 01:11:47.816919 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.816900 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e01bda7-1fb3-43e0-9197-f5627fd71fd8" containerName="test-curl-7469-139661443684656" Jul 17 01:11:47.817011 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.816922 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e01bda7-1fb3-43e0-9197-f5627fd71fd8" containerName="test-curl-7469-139661443684656" Jul 17 01:11:47.817072 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.817064 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e01bda7-1fb3-43e0-9197-f5627fd71fd8" containerName="test-curl-7469-139661443684656" Jul 17 01:11:47.820299 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.820277 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:47.823034 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.823002 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czq5d\"" Jul 17 01:11:47.826171 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.826148 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443547760"] Jul 17 01:11:47.828159 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.828138 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dmw\" (UniqueName: \"kubernetes.io/projected/1042ca36-cfbe-4a18-b9c8-e9f8e9909c69-kube-api-access-86dmw\") pod \"test-curl-7469-139661443547760\" (UID: \"1042ca36-cfbe-4a18-b9c8-e9f8e9909c69\") " pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:47.928951 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.928881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dmw\" (UniqueName: \"kubernetes.io/projected/1042ca36-cfbe-4a18-b9c8-e9f8e9909c69-kube-api-access-86dmw\") pod \"test-curl-7469-139661443547760\" (UID: \"1042ca36-cfbe-4a18-b9c8-e9f8e9909c69\") " pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:47.936741 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:47.936710 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dmw\" (UniqueName: \"kubernetes.io/projected/1042ca36-cfbe-4a18-b9c8-e9f8e9909c69-kube-api-access-86dmw\") pod \"test-curl-7469-139661443547760\" (UID: \"1042ca36-cfbe-4a18-b9c8-e9f8e9909c69\") " pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:48.130298 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:48.130272 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:48.253464 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:48.253441 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443547760"] Jul 17 01:11:48.255633 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:11:48.255599 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1042ca36_cfbe_4a18_b9c8_e9f8e9909c69.slice/crio-5750bb2ff43073f5933141fa06934eaff208f08dc0996bad0a711ef3047d215b WatchSource:0}: Error finding container 5750bb2ff43073f5933141fa06934eaff208f08dc0996bad0a711ef3047d215b: Status 404 returned error can't find the container with id 5750bb2ff43073f5933141fa06934eaff208f08dc0996bad0a711ef3047d215b Jul 17 01:11:49.062522 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:49.062489 2572 generic.go:356] "Generic (PLEG): container finished" podID="1042ca36-cfbe-4a18-b9c8-e9f8e9909c69" containerID="407437fc4e99b7cd669a27c194219485e9d5eef5011976735ae966cb2183be1f" exitCode=0 Jul 17 01:11:49.062945 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:49.062562 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443547760" event={"ID":"1042ca36-cfbe-4a18-b9c8-e9f8e9909c69","Type":"ContainerDied","Data":"407437fc4e99b7cd669a27c194219485e9d5eef5011976735ae966cb2183be1f"} Jul 17 01:11:49.062945 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:49.062588 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443547760" event={"ID":"1042ca36-cfbe-4a18-b9c8-e9f8e9909c69","Type":"ContainerStarted","Data":"5750bb2ff43073f5933141fa06934eaff208f08dc0996bad0a711ef3047d215b"} Jul 17 01:11:49.544395 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:49.544360 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:11:50.204571 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.204542 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:50.223238 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.223193 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7469-139661443547760_1042ca36-cfbe-4a18-b9c8-e9f8e9909c69/test-curl-7469-139661443547760/0.log" Jul 17 01:11:50.245134 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.245082 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7469-139661443547760"] Jul 17 01:11:50.248358 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.248328 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7469-139661443547760"] Jul 17 01:11:50.256763 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.256738 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86dmw\" (UniqueName: \"kubernetes.io/projected/1042ca36-cfbe-4a18-b9c8-e9f8e9909c69-kube-api-access-86dmw\") pod \"1042ca36-cfbe-4a18-b9c8-e9f8e9909c69\" (UID: \"1042ca36-cfbe-4a18-b9c8-e9f8e9909c69\") " Jul 17 01:11:50.258824 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.258792 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1042ca36-cfbe-4a18-b9c8-e9f8e9909c69-kube-api-access-86dmw" (OuterVolumeSpecName: "kube-api-access-86dmw") pod "1042ca36-cfbe-4a18-b9c8-e9f8e9909c69" (UID: "1042ca36-cfbe-4a18-b9c8-e9f8e9909c69"). InnerVolumeSpecName "kube-api-access-86dmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:11:50.358295 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.358262 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86dmw\" (UniqueName: \"kubernetes.io/projected/1042ca36-cfbe-4a18-b9c8-e9f8e9909c69-kube-api-access-86dmw\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:50.366750 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.366722 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:50.367126 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.367112 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1042ca36-cfbe-4a18-b9c8-e9f8e9909c69" containerName="test-curl-7469-139661443547760" Jul 17 01:11:50.367186 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.367128 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1042ca36-cfbe-4a18-b9c8-e9f8e9909c69" containerName="test-curl-7469-139661443547760" Jul 17 01:11:50.367223 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.367198 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1042ca36-cfbe-4a18-b9c8-e9f8e9909c69" containerName="test-curl-7469-139661443547760" Jul 17 01:11:50.370124 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.370108 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:50.373529 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.373502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:50.458940 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.458880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqsrj\" (UniqueName: \"kubernetes.io/projected/c90c3864-7bda-4dab-a93e-4683dc109331-kube-api-access-fqsrj\") pod \"test-curl-7469-139661443548768\" (UID: \"c90c3864-7bda-4dab-a93e-4683dc109331\") " pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:50.559584 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.559538 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqsrj\" (UniqueName: \"kubernetes.io/projected/c90c3864-7bda-4dab-a93e-4683dc109331-kube-api-access-fqsrj\") pod \"test-curl-7469-139661443548768\" (UID: \"c90c3864-7bda-4dab-a93e-4683dc109331\") " pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:50.566963 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.566943 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqsrj\" (UniqueName: \"kubernetes.io/projected/c90c3864-7bda-4dab-a93e-4683dc109331-kube-api-access-fqsrj\") pod \"test-curl-7469-139661443548768\" (UID: \"c90c3864-7bda-4dab-a93e-4683dc109331\") " pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:50.682439 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:50.682355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:51.012375 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.012324 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:51.014582 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:11:51.014553 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc90c3864_7bda_4dab_a93e_4683dc109331.slice/crio-50e0db7ebb7142df2538ac42f6f7dcc450a23b5a14f50d19e950305bf4b52ff4 WatchSource:0}: Error finding container 50e0db7ebb7142df2538ac42f6f7dcc450a23b5a14f50d19e950305bf4b52ff4: Status 404 returned error can't find the container with id 50e0db7ebb7142df2538ac42f6f7dcc450a23b5a14f50d19e950305bf4b52ff4 Jul 17 01:11:51.074398 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.074359 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443548768" event={"ID":"c90c3864-7bda-4dab-a93e-4683dc109331","Type":"ContainerStarted","Data":"50e0db7ebb7142df2538ac42f6f7dcc450a23b5a14f50d19e950305bf4b52ff4"} Jul 17 01:11:51.075481 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.075462 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5750bb2ff43073f5933141fa06934eaff208f08dc0996bad0a711ef3047d215b" Jul 17 01:11:51.075481 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.075471 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:51.085369 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:11:51.085345 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7469-139661443547760\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="1042ca36-cfbe-4a18-b9c8-e9f8e9909c69" pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:51.374206 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.374154 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" podUID="b0bddca1-6d0f-46a1-9bfa-df0b8594b439" containerName="main" containerID="cri-o://134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2" gracePeriod=45 Jul 17 01:11:51.609460 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.609437 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:11:51.629278 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:11:51.629198 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7469-139661443547760\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="1042ca36-cfbe-4a18-b9c8-e9f8e9909c69" pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:51.669532 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.669501 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tls-certs\") pod \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " Jul 17 01:11:51.669532 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.669531 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tmp-dir\") pod \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " Jul 17 01:11:51.669780 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.669563 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-model-cache\") pod \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " Jul 17 01:11:51.669780 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.669653 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-home\") pod \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " Jul 17 01:11:51.669780 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.669695 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-dshm\") pod \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " Jul 17 01:11:51.669780 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.669728 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs8zj\" (UniqueName: \"kubernetes.io/projected/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-kube-api-access-bs8zj\") pod \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\" (UID: \"b0bddca1-6d0f-46a1-9bfa-df0b8594b439\") " Jul 17 01:11:51.670008 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.669795 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0bddca1-6d0f-46a1-9bfa-df0b8594b439" (UID: "b0bddca1-6d0f-46a1-9bfa-df0b8594b439"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:11:51.670008 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.669813 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0bddca1-6d0f-46a1-9bfa-df0b8594b439" (UID: "b0bddca1-6d0f-46a1-9bfa-df0b8594b439"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:11:51.670008 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.669944 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-home" (OuterVolumeSpecName: "home") pod "b0bddca1-6d0f-46a1-9bfa-df0b8594b439" (UID: "b0bddca1-6d0f-46a1-9bfa-df0b8594b439"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:11:51.670156 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.670035 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:51.670156 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.670048 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:51.670156 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.670057 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:51.671695 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.671673 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-dshm" (OuterVolumeSpecName: "dshm") pod "b0bddca1-6d0f-46a1-9bfa-df0b8594b439" (UID: "b0bddca1-6d0f-46a1-9bfa-df0b8594b439"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:11:51.671776 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.671686 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-kube-api-access-bs8zj" (OuterVolumeSpecName: "kube-api-access-bs8zj") pod "b0bddca1-6d0f-46a1-9bfa-df0b8594b439" (UID: "b0bddca1-6d0f-46a1-9bfa-df0b8594b439"). InnerVolumeSpecName "kube-api-access-bs8zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:11:51.671776 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.671705 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0bddca1-6d0f-46a1-9bfa-df0b8594b439" (UID: "b0bddca1-6d0f-46a1-9bfa-df0b8594b439"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:11:51.771277 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.771245 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:51.771277 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.771272 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8zj\" (UniqueName: \"kubernetes.io/projected/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-kube-api-access-bs8zj\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:51.771277 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:51.771283 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bddca1-6d0f-46a1-9bfa-df0b8594b439-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:52.082720 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.082636 2572 generic.go:356] "Generic (PLEG): container finished" podID="c90c3864-7bda-4dab-a93e-4683dc109331" containerID="4cae672ee5913214d23b4122c6291b1e8d73d6321e5563d6f5a45027601bd118" exitCode=0 Jul 17 01:11:52.084134 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.084103 2572 generic.go:356] "Generic (PLEG): container finished" podID="b0bddca1-6d0f-46a1-9bfa-df0b8594b439" containerID="134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2" exitCode=0 Jul 17 01:11:52.084282 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.084163 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" Jul 17 01:11:52.085074 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:11:52.085044 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7469-139661443547760\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="1042ca36-cfbe-4a18-b9c8-e9f8e9909c69" pod="opendatahub/test-curl-7469-139661443547760" Jul 17 01:11:52.088642 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.088615 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1042ca36-cfbe-4a18-b9c8-e9f8e9909c69" path="/var/lib/kubelet/pods/1042ca36-cfbe-4a18-b9c8-e9f8e9909c69/volumes" Jul 17 01:11:52.089174 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.089152 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443548768" event={"ID":"c90c3864-7bda-4dab-a93e-4683dc109331","Type":"ContainerDied","Data":"4cae672ee5913214d23b4122c6291b1e8d73d6321e5563d6f5a45027601bd118"} Jul 17 01:11:52.089233 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.089193 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" event={"ID":"b0bddca1-6d0f-46a1-9bfa-df0b8594b439","Type":"ContainerDied","Data":"134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2"} Jul 17 01:11:52.089233 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.089212 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp" event={"ID":"b0bddca1-6d0f-46a1-9bfa-df0b8594b439","Type":"ContainerDied","Data":"4d4cf74a09eb00f9e88bf34d82ed56b9fc72f2de66bba443a0bacb1c9eac7402"} Jul 17 01:11:52.089335 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.089234 2572 scope.go:117] "RemoveContainer" containerID="134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2" Jul 17 01:11:52.098928 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.098906 2572 scope.go:117] "RemoveContainer" containerID="134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2" Jul 17 01:11:52.099221 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:11:52.099203 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2\": container with ID starting with 134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2 not found: ID does not exist" containerID="134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2" Jul 17 01:11:52.099293 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.099231 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2"} err="failed to get container status \"134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2\": rpc error: code = NotFound desc = could not find container \"134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2\": container with ID starting with 134b6e3b715596692f959038d46e8b254133c9d83295e87cfb67784e708a6ae2 not found: ID does not exist" Jul 17 01:11:52.110688 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.110661 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp"] Jul 17 01:11:52.112213 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.112195 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e1dd24/rate-test-model-e1dd24-kserve-68fddbf59f-vwtqp"] Jul 17 01:11:52.324882 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.324841 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" podUID="8a50c22b-c87e-4be3-96b9-155f3491663b" containerName="main" containerID="cri-o://35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c" gracePeriod=45 Jul 17 01:11:52.570817 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.570794 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:11:52.679499 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679468 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wsj7\" (UniqueName: \"kubernetes.io/projected/8a50c22b-c87e-4be3-96b9-155f3491663b-kube-api-access-4wsj7\") pod \"8a50c22b-c87e-4be3-96b9-155f3491663b\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " Jul 17 01:11:52.679666 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679517 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50c22b-c87e-4be3-96b9-155f3491663b-tls-certs\") pod \"8a50c22b-c87e-4be3-96b9-155f3491663b\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " Jul 17 01:11:52.679666 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679535 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-dshm\") pod \"8a50c22b-c87e-4be3-96b9-155f3491663b\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " Jul 17 01:11:52.679666 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679561 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-model-cache\") pod \"8a50c22b-c87e-4be3-96b9-155f3491663b\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " Jul 17 01:11:52.679666 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679597 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-home\") pod \"8a50c22b-c87e-4be3-96b9-155f3491663b\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " Jul 17 01:11:52.679666 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679621 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-tmp-dir\") pod \"8a50c22b-c87e-4be3-96b9-155f3491663b\" (UID: \"8a50c22b-c87e-4be3-96b9-155f3491663b\") " Jul 17 01:11:52.679961 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679833 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-home" (OuterVolumeSpecName: "home") pod "8a50c22b-c87e-4be3-96b9-155f3491663b" (UID: "8a50c22b-c87e-4be3-96b9-155f3491663b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:11:52.679961 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679846 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a50c22b-c87e-4be3-96b9-155f3491663b" (UID: "8a50c22b-c87e-4be3-96b9-155f3491663b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:11:52.679961 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679949 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:52.680092 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679966 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:52.680092 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.679987 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a50c22b-c87e-4be3-96b9-155f3491663b" (UID: "8a50c22b-c87e-4be3-96b9-155f3491663b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:11:52.681680 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.681658 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-dshm" (OuterVolumeSpecName: "dshm") pod "8a50c22b-c87e-4be3-96b9-155f3491663b" (UID: "8a50c22b-c87e-4be3-96b9-155f3491663b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:11:52.681776 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.681654 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a50c22b-c87e-4be3-96b9-155f3491663b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a50c22b-c87e-4be3-96b9-155f3491663b" (UID: "8a50c22b-c87e-4be3-96b9-155f3491663b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:11:52.681776 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.681711 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a50c22b-c87e-4be3-96b9-155f3491663b-kube-api-access-4wsj7" (OuterVolumeSpecName: "kube-api-access-4wsj7") pod "8a50c22b-c87e-4be3-96b9-155f3491663b" (UID: "8a50c22b-c87e-4be3-96b9-155f3491663b"). InnerVolumeSpecName "kube-api-access-4wsj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:11:52.780362 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.780327 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wsj7\" (UniqueName: \"kubernetes.io/projected/8a50c22b-c87e-4be3-96b9-155f3491663b-kube-api-access-4wsj7\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:52.780362 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.780356 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50c22b-c87e-4be3-96b9-155f3491663b-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:52.780362 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.780366 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:52.780584 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:52.780374 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50c22b-c87e-4be3-96b9-155f3491663b-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:53.091199 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.091107 2572 generic.go:356] "Generic (PLEG): container finished" podID="8a50c22b-c87e-4be3-96b9-155f3491663b" containerID="35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c" exitCode=0 Jul 17 01:11:53.091199 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.091149 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" Jul 17 01:11:53.091199 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.091158 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" event={"ID":"8a50c22b-c87e-4be3-96b9-155f3491663b","Type":"ContainerDied","Data":"35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c"} Jul 17 01:11:53.091199 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.091196 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8" event={"ID":"8a50c22b-c87e-4be3-96b9-155f3491663b","Type":"ContainerDied","Data":"e4c33ddc689470f37e8cfaa32ed75f972da6624479fb4efb36c0e8cd2eee799f"} Jul 17 01:11:53.091523 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.091211 2572 scope.go:117] "RemoveContainer" containerID="35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c" Jul 17 01:11:53.100683 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.100654 2572 scope.go:117] "RemoveContainer" containerID="35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c" Jul 17 01:11:53.100970 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:11:53.100949 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c\": container with ID starting with 35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c not found: ID does not exist" containerID="35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c" Jul 17 01:11:53.101047 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.100977 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c"} err="failed to get container status \"35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c\": rpc error: code = NotFound desc = could not find container \"35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c\": container with ID starting with 35c95262c8ac18aa85b9ad81aa7b11a390c5d45bd169a084d3b83e362fc3416c not found: ID does not exist" Jul 17 01:11:53.111347 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.111325 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8"] Jul 17 01:11:53.114717 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.114695 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ac54e2/rate-test-model-ac54e2-kserve-7d6cd457f5-2dbp8"] Jul 17 01:11:53.227381 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.227358 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:53.247478 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.247445 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7469-139661443548768_c90c3864-7bda-4dab-a93e-4683dc109331/test-curl-7469-139661443548768/0.log" Jul 17 01:11:53.267966 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.267933 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:53.272906 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.272864 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:53.283487 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.283465 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqsrj\" (UniqueName: \"kubernetes.io/projected/c90c3864-7bda-4dab-a93e-4683dc109331-kube-api-access-fqsrj\") pod \"c90c3864-7bda-4dab-a93e-4683dc109331\" (UID: \"c90c3864-7bda-4dab-a93e-4683dc109331\") " Jul 17 01:11:53.285630 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.285610 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90c3864-7bda-4dab-a93e-4683dc109331-kube-api-access-fqsrj" (OuterVolumeSpecName: "kube-api-access-fqsrj") pod "c90c3864-7bda-4dab-a93e-4683dc109331" (UID: "c90c3864-7bda-4dab-a93e-4683dc109331"). InnerVolumeSpecName "kube-api-access-fqsrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:11:53.385092 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.385060 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqsrj\" (UniqueName: \"kubernetes.io/projected/c90c3864-7bda-4dab-a93e-4683dc109331-kube-api-access-fqsrj\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:53.395846 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.395815 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:53.396209 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.396196 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0bddca1-6d0f-46a1-9bfa-df0b8594b439" containerName="main" Jul 17 01:11:53.396251 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.396211 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0bddca1-6d0f-46a1-9bfa-df0b8594b439" containerName="main" Jul 17 01:11:53.396251 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.396219 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a50c22b-c87e-4be3-96b9-155f3491663b" containerName="main" Jul 17 01:11:53.396251 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.396225 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a50c22b-c87e-4be3-96b9-155f3491663b" containerName="main" Jul 17 01:11:53.396251 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.396239 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90c3864-7bda-4dab-a93e-4683dc109331" containerName="test-curl-7469-139661443548768" Jul 17 01:11:53.396251 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.396245 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90c3864-7bda-4dab-a93e-4683dc109331" containerName="test-curl-7469-139661443548768" Jul 17 01:11:53.396404 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.396299 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90c3864-7bda-4dab-a93e-4683dc109331" containerName="test-curl-7469-139661443548768" Jul 17 01:11:53.396404 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.396308 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a50c22b-c87e-4be3-96b9-155f3491663b" containerName="main" Jul 17 01:11:53.396404 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.396317 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0bddca1-6d0f-46a1-9bfa-df0b8594b439" containerName="main" Jul 17 01:11:53.399370 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.399356 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:53.404875 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.404845 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:53.486028 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.485996 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k622b\" (UniqueName: \"kubernetes.io/projected/dc88b87d-cf3d-47a2-896b-bad71e98e177-kube-api-access-k622b\") pod \"test-curl-7469-139661443548768\" (UID: \"dc88b87d-cf3d-47a2-896b-bad71e98e177\") " pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:53.586521 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.586488 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k622b\" (UniqueName: \"kubernetes.io/projected/dc88b87d-cf3d-47a2-896b-bad71e98e177-kube-api-access-k622b\") pod \"test-curl-7469-139661443548768\" (UID: \"dc88b87d-cf3d-47a2-896b-bad71e98e177\") " pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:53.594428 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.594400 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k622b\" (UniqueName: \"kubernetes.io/projected/dc88b87d-cf3d-47a2-896b-bad71e98e177-kube-api-access-k622b\") pod \"test-curl-7469-139661443548768\" (UID: \"dc88b87d-cf3d-47a2-896b-bad71e98e177\") " pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:53.710150 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.710046 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:53.831322 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:53.831298 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:53.833568 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:11:53.833535 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc88b87d_cf3d_47a2_896b_bad71e98e177.slice/crio-6ca5f39f8fc8d8f3448226147797eedcd72afbefe550788c2876c732445d35dc WatchSource:0}: Error finding container 6ca5f39f8fc8d8f3448226147797eedcd72afbefe550788c2876c732445d35dc: Status 404 returned error can't find the container with id 6ca5f39f8fc8d8f3448226147797eedcd72afbefe550788c2876c732445d35dc Jul 17 01:11:54.088739 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:54.088659 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a50c22b-c87e-4be3-96b9-155f3491663b" path="/var/lib/kubelet/pods/8a50c22b-c87e-4be3-96b9-155f3491663b/volumes" Jul 17 01:11:54.089095 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:54.089080 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0bddca1-6d0f-46a1-9bfa-df0b8594b439" path="/var/lib/kubelet/pods/b0bddca1-6d0f-46a1-9bfa-df0b8594b439/volumes" Jul 17 01:11:54.089450 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:54.089438 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c90c3864-7bda-4dab-a93e-4683dc109331" path="/var/lib/kubelet/pods/c90c3864-7bda-4dab-a93e-4683dc109331/volumes" Jul 17 01:11:54.096268 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:54.096241 2572 scope.go:117] "RemoveContainer" containerID="4cae672ee5913214d23b4122c6291b1e8d73d6321e5563d6f5a45027601bd118" Jul 17 01:11:54.096414 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:54.096249 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:54.098530 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:54.098503 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443548768" event={"ID":"dc88b87d-cf3d-47a2-896b-bad71e98e177","Type":"ContainerStarted","Data":"6ca5f39f8fc8d8f3448226147797eedcd72afbefe550788c2876c732445d35dc"} Jul 17 01:11:55.105011 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:55.104979 2572 generic.go:356] "Generic (PLEG): container finished" podID="dc88b87d-cf3d-47a2-896b-bad71e98e177" containerID="f8eb19d97361c064bb41ae6e18175ccdccc0ca323f712146aace2d2f6c0385d5" exitCode=0 Jul 17 01:11:55.105451 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:55.105070 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7469-139661443548768" event={"ID":"dc88b87d-cf3d-47a2-896b-bad71e98e177","Type":"ContainerDied","Data":"f8eb19d97361c064bb41ae6e18175ccdccc0ca323f712146aace2d2f6c0385d5"} Jul 17 01:11:56.240461 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.240434 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:56.258317 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.258290 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7469-139661443548768_dc88b87d-cf3d-47a2-896b-bad71e98e177/test-curl-7469-139661443548768/0.log" Jul 17 01:11:56.280970 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.280933 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:56.283932 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.283901 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7469-139661443548768"] Jul 17 01:11:56.311418 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.311387 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k622b\" (UniqueName: \"kubernetes.io/projected/dc88b87d-cf3d-47a2-896b-bad71e98e177-kube-api-access-k622b\") pod \"dc88b87d-cf3d-47a2-896b-bad71e98e177\" (UID: \"dc88b87d-cf3d-47a2-896b-bad71e98e177\") " Jul 17 01:11:56.313407 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.313386 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc88b87d-cf3d-47a2-896b-bad71e98e177-kube-api-access-k622b" (OuterVolumeSpecName: "kube-api-access-k622b") pod "dc88b87d-cf3d-47a2-896b-bad71e98e177" (UID: "dc88b87d-cf3d-47a2-896b-bad71e98e177"). InnerVolumeSpecName "kube-api-access-k622b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:11:56.412043 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.411965 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k622b\" (UniqueName: \"kubernetes.io/projected/dc88b87d-cf3d-47a2-896b-bad71e98e177-kube-api-access-k622b\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:56.448396 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.448362 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q"] Jul 17 01:11:56.448806 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.448789 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc88b87d-cf3d-47a2-896b-bad71e98e177" containerName="test-curl-7469-139661443548768" Jul 17 01:11:56.448920 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.448808 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc88b87d-cf3d-47a2-896b-bad71e98e177" containerName="test-curl-7469-139661443548768" Jul 17 01:11:56.448920 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.448916 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc88b87d-cf3d-47a2-896b-bad71e98e177" containerName="test-curl-7469-139661443548768" Jul 17 01:11:56.451786 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.451767 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" Jul 17 01:11:56.454089 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.454070 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfd9x\"" Jul 17 01:11:56.454165 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.454077 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:11:56.459557 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.459523 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q"] Jul 17 01:11:56.512394 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.512354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7143ca6-5d37-404d-bb13-442437242380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q\" (UID: \"d7143ca6-5d37-404d-bb13-442437242380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" Jul 17 01:11:56.613780 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.613747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7143ca6-5d37-404d-bb13-442437242380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q\" (UID: \"d7143ca6-5d37-404d-bb13-442437242380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" Jul 17 01:11:56.614353 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.614335 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7143ca6-5d37-404d-bb13-442437242380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q\" (UID: \"d7143ca6-5d37-404d-bb13-442437242380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" Jul 17 01:11:56.763063 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.762981 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" Jul 17 01:11:56.887647 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.887614 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q"] Jul 17 01:11:56.890733 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:11:56.890705 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7143ca6_5d37_404d_bb13_442437242380.slice/crio-4cd1d2374e893321bfb1599ba16675f21a2b17d35a134384ffbd1cf6a101f418 WatchSource:0}: Error finding container 4cd1d2374e893321bfb1599ba16675f21a2b17d35a134384ffbd1cf6a101f418: Status 404 returned error can't find the container with id 4cd1d2374e893321bfb1599ba16675f21a2b17d35a134384ffbd1cf6a101f418 Jul 17 01:11:56.953069 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:56.953038 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:11:57.053262 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:57.053168 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:11:57.118082 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:57.118044 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" event={"ID":"d7143ca6-5d37-404d-bb13-442437242380","Type":"ContainerStarted","Data":"4cd1d2374e893321bfb1599ba16675f21a2b17d35a134384ffbd1cf6a101f418"} Jul 17 01:11:57.119184 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:57.119166 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:57.119289 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:57.119213 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ca5f39f8fc8d8f3448226147797eedcd72afbefe550788c2876c732445d35dc" Jul 17 01:11:57.128538 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:11:57.128514 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7469-139661443548768\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="dc88b87d-cf3d-47a2-896b-bad71e98e177" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:58.086507 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:11:58.086475 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7469-139661443548768\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="dc88b87d-cf3d-47a2-896b-bad71e98e177" pod="opendatahub/test-curl-7469-139661443548768" Jul 17 01:11:58.088651 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:58.088627 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc88b87d-cf3d-47a2-896b-bad71e98e177" path="/var/lib/kubelet/pods/dc88b87d-cf3d-47a2-896b-bad71e98e177/volumes" Jul 17 01:11:58.125389 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:58.125354 2572 generic.go:356] "Generic (PLEG): container finished" podID="d7143ca6-5d37-404d-bb13-442437242380" containerID="331ce33ecf31249ecc8d862f39710f9e86f622f1a2c21be2cee5948e7910dbf7" exitCode=0 Jul 17 01:11:58.125492 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:58.125440 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" event={"ID":"d7143ca6-5d37-404d-bb13-442437242380","Type":"ContainerDied","Data":"331ce33ecf31249ecc8d862f39710f9e86f622f1a2c21be2cee5948e7910dbf7"} Jul 17 01:11:59.261455 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.261429 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" Jul 17 01:11:59.337417 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.337381 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7143ca6-5d37-404d-bb13-442437242380-maas-api-service-ca\") pod \"d7143ca6-5d37-404d-bb13-442437242380\" (UID: \"d7143ca6-5d37-404d-bb13-442437242380\") " Jul 17 01:11:59.337784 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.337758 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7143ca6-5d37-404d-bb13-442437242380-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7143ca6-5d37-404d-bb13-442437242380" (UID: "d7143ca6-5d37-404d-bb13-442437242380"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:11:59.438796 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.438714 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7143ca6-5d37-404d-bb13-442437242380-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:11:59.454320 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.454289 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f"] Jul 17 01:11:59.454658 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.454646 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7143ca6-5d37-404d-bb13-442437242380" containerName="revoke-keys" Jul 17 01:11:59.454705 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.454659 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7143ca6-5d37-404d-bb13-442437242380" containerName="revoke-keys" Jul 17 01:11:59.454740 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.454712 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7143ca6-5d37-404d-bb13-442437242380" containerName="revoke-keys" Jul 17 01:11:59.457815 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.457796 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" Jul 17 01:11:59.463831 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.463810 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f"] Jul 17 01:11:59.539324 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.539287 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745c2ea2-8c3e-466f-973e-d7536392cc1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-z242f\" (UID: \"745c2ea2-8c3e-466f-973e-d7536392cc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" Jul 17 01:11:59.640340 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.640302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745c2ea2-8c3e-466f-973e-d7536392cc1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-z242f\" (UID: \"745c2ea2-8c3e-466f-973e-d7536392cc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" Jul 17 01:11:59.640972 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.640954 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745c2ea2-8c3e-466f-973e-d7536392cc1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4692a03f-z242f\" (UID: \"745c2ea2-8c3e-466f-973e-d7536392cc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" Jul 17 01:11:59.769077 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.768989 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" Jul 17 01:11:59.901503 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:11:59.901478 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f"] Jul 17 01:11:59.904109 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:11:59.904083 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745c2ea2_8c3e_466f_973e_d7536392cc1c.slice/crio-272cb1372a0b4c820b16dc79b9e9e972fc5988cb23748cacf331be4656aa6fa7 WatchSource:0}: Error finding container 272cb1372a0b4c820b16dc79b9e9e972fc5988cb23748cacf331be4656aa6fa7: Status 404 returned error can't find the container with id 272cb1372a0b4c820b16dc79b9e9e972fc5988cb23748cacf331be4656aa6fa7 Jul 17 01:12:00.135802 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:00.135767 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" event={"ID":"745c2ea2-8c3e-466f-973e-d7536392cc1c","Type":"ContainerStarted","Data":"272cb1372a0b4c820b16dc79b9e9e972fc5988cb23748cacf331be4656aa6fa7"} Jul 17 01:12:00.137054 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:00.137033 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" Jul 17 01:12:00.137054 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:00.137041 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q" event={"ID":"d7143ca6-5d37-404d-bb13-442437242380","Type":"ContainerDied","Data":"4cd1d2374e893321bfb1599ba16675f21a2b17d35a134384ffbd1cf6a101f418"} Jul 17 01:12:00.137211 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:00.137083 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cd1d2374e893321bfb1599ba16675f21a2b17d35a134384ffbd1cf6a101f418" Jul 17 01:12:01.143553 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.143515 2572 generic.go:356] "Generic (PLEG): container finished" podID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerID="56ae1280cc506cde751090fea2417fcff05183060837e177e4df815d7ee3c862" exitCode=6 Jul 17 01:12:01.143948 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.143612 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" event={"ID":"745c2ea2-8c3e-466f-973e-d7536392cc1c","Type":"ContainerDied","Data":"56ae1280cc506cde751090fea2417fcff05183060837e177e4df815d7ee3c862"} Jul 17 01:12:01.143948 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.143751 2572 scope.go:117] "RemoveContainer" containerID="56ae1280cc506cde751090fea2417fcff05183060837e177e4df815d7ee3c862" Jul 17 01:12:01.309816 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.309786 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9"] Jul 17 01:12:01.314341 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.314324 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" Jul 17 01:12:01.319752 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.319724 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9"] Jul 17 01:12:01.355917 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.355867 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae818f-c443-45bc-8bb0-b968deb2879d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9\" (UID: \"d0ae818f-c443-45bc-8bb0-b968deb2879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" Jul 17 01:12:01.457013 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.456931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae818f-c443-45bc-8bb0-b968deb2879d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9\" (UID: \"d0ae818f-c443-45bc-8bb0-b968deb2879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" Jul 17 01:12:01.457524 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.457506 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae818f-c443-45bc-8bb0-b968deb2879d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9\" (UID: \"d0ae818f-c443-45bc-8bb0-b968deb2879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" Jul 17 01:12:01.625748 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.625708 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" Jul 17 01:12:01.753360 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:01.753266 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9"] Jul 17 01:12:01.755540 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:12:01.755509 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ae818f_c443_45bc_8bb0_b968deb2879d.slice/crio-393e6b0184e00c7231a6941946b323950c1eb02b97c706b3e6e6884dd7c39786 WatchSource:0}: Error finding container 393e6b0184e00c7231a6941946b323950c1eb02b97c706b3e6e6884dd7c39786: Status 404 returned error can't find the container with id 393e6b0184e00c7231a6941946b323950c1eb02b97c706b3e6e6884dd7c39786 Jul 17 01:12:02.150087 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:02.150046 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" event={"ID":"d0ae818f-c443-45bc-8bb0-b968deb2879d","Type":"ContainerStarted","Data":"3c16394c32e493f5bea8b6a5815800bf455d89836e1ade045b1a143028431b6c"} Jul 17 01:12:02.150522 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:02.150115 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" event={"ID":"d0ae818f-c443-45bc-8bb0-b968deb2879d","Type":"ContainerStarted","Data":"393e6b0184e00c7231a6941946b323950c1eb02b97c706b3e6e6884dd7c39786"} Jul 17 01:12:02.151801 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:02.151777 2572 generic.go:356] "Generic (PLEG): container finished" podID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerID="255923224d1dea0505366251846547f5a10ea6ba89f44ba05735fc490ed47617" exitCode=6 Jul 17 01:12:02.151936 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:02.151814 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" event={"ID":"745c2ea2-8c3e-466f-973e-d7536392cc1c","Type":"ContainerDied","Data":"255923224d1dea0505366251846547f5a10ea6ba89f44ba05735fc490ed47617"} Jul 17 01:12:02.151936 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:02.151855 2572 scope.go:117] "RemoveContainer" containerID="56ae1280cc506cde751090fea2417fcff05183060837e177e4df815d7ee3c862" Jul 17 01:12:02.152095 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:02.152079 2572 scope.go:117] "RemoveContainer" containerID="255923224d1dea0505366251846547f5a10ea6ba89f44ba05735fc490ed47617" Jul 17 01:12:02.152300 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:12:02.152280 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-z242f_odh-ai-gateway-infra(745c2ea2-8c3e-466f-973e-d7536392cc1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" Jul 17 01:12:02.165175 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:02.165103 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" podStartSLOduration=1.16508938 podStartE2EDuration="1.16508938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:12:02.163507293 +0000 UTC m=+3294.910016328" watchObservedRunningTime="2026-07-17 01:12:02.16508938 +0000 UTC m=+3294.911598392" Jul 17 01:12:03.158867 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.158827 2572 generic.go:356] "Generic (PLEG): container finished" podID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerID="3c16394c32e493f5bea8b6a5815800bf455d89836e1ade045b1a143028431b6c" exitCode=6 Jul 17 01:12:03.159327 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.158873 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" event={"ID":"d0ae818f-c443-45bc-8bb0-b968deb2879d","Type":"ContainerDied","Data":"3c16394c32e493f5bea8b6a5815800bf455d89836e1ade045b1a143028431b6c"} Jul 17 01:12:03.159327 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.159137 2572 scope.go:117] "RemoveContainer" containerID="3c16394c32e493f5bea8b6a5815800bf455d89836e1ade045b1a143028431b6c" Jul 17 01:12:03.160717 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.160703 2572 scope.go:117] "RemoveContainer" containerID="255923224d1dea0505366251846547f5a10ea6ba89f44ba05735fc490ed47617" Jul 17 01:12:03.160871 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:12:03.160855 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-z242f_odh-ai-gateway-infra(745c2ea2-8c3e-466f-973e-d7536392cc1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" Jul 17 01:12:03.731531 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.731494 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt"] Jul 17 01:12:03.734851 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.734835 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" Jul 17 01:12:03.743088 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.742866 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt"] Jul 17 01:12:03.776860 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.776826 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b90b29-d87f-4d73-a80e-e771995f7d98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt\" (UID: \"27b90b29-d87f-4d73-a80e-e771995f7d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" Jul 17 01:12:03.878233 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.878181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b90b29-d87f-4d73-a80e-e771995f7d98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt\" (UID: \"27b90b29-d87f-4d73-a80e-e771995f7d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" Jul 17 01:12:03.878820 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:03.878799 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b90b29-d87f-4d73-a80e-e771995f7d98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt\" (UID: \"27b90b29-d87f-4d73-a80e-e771995f7d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" Jul 17 01:12:04.055113 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:04.055022 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" Jul 17 01:12:04.169590 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:04.169558 2572 generic.go:356] "Generic (PLEG): container finished" podID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerID="ee2006f825fec86e1ff38f9278423cd2f41ffca8e5212ea22aab7355fc7bd6d9" exitCode=6 Jul 17 01:12:04.170049 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:04.169658 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" event={"ID":"d0ae818f-c443-45bc-8bb0-b968deb2879d","Type":"ContainerDied","Data":"ee2006f825fec86e1ff38f9278423cd2f41ffca8e5212ea22aab7355fc7bd6d9"} Jul 17 01:12:04.170049 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:04.169738 2572 scope.go:117] "RemoveContainer" containerID="3c16394c32e493f5bea8b6a5815800bf455d89836e1ade045b1a143028431b6c" Jul 17 01:12:04.170049 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:04.170046 2572 scope.go:117] "RemoveContainer" containerID="ee2006f825fec86e1ff38f9278423cd2f41ffca8e5212ea22aab7355fc7bd6d9" Jul 17 01:12:04.170269 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:12:04.170251 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9_odh-ai-gateway-infra(d0ae818f-c443-45bc-8bb0-b968deb2879d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" Jul 17 01:12:04.181729 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:04.181710 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt"] Jul 17 01:12:04.184852 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:12:04.184832 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b90b29_d87f_4d73_a80e_e771995f7d98.slice/crio-5e82e49b1d1f6326904a414d3059b368892789e32d694965c2aba24e9325c624 WatchSource:0}: Error finding container 5e82e49b1d1f6326904a414d3059b368892789e32d694965c2aba24e9325c624: Status 404 returned error can't find the container with id 5e82e49b1d1f6326904a414d3059b368892789e32d694965c2aba24e9325c624 Jul 17 01:12:05.176298 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:05.176261 2572 generic.go:356] "Generic (PLEG): container finished" podID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerID="c7dd3ca5e6feead448dabe7cb87a1cb186137288ab2c41609419b1622078ef3d" exitCode=6 Jul 17 01:12:05.176752 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:05.176350 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" event={"ID":"27b90b29-d87f-4d73-a80e-e771995f7d98","Type":"ContainerDied","Data":"c7dd3ca5e6feead448dabe7cb87a1cb186137288ab2c41609419b1622078ef3d"} Jul 17 01:12:05.176752 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:05.176409 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" event={"ID":"27b90b29-d87f-4d73-a80e-e771995f7d98","Type":"ContainerStarted","Data":"5e82e49b1d1f6326904a414d3059b368892789e32d694965c2aba24e9325c624"} Jul 17 01:12:05.176752 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:05.176494 2572 scope.go:117] "RemoveContainer" containerID="c7dd3ca5e6feead448dabe7cb87a1cb186137288ab2c41609419b1622078ef3d" Jul 17 01:12:05.178066 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:05.178049 2572 scope.go:117] "RemoveContainer" containerID="ee2006f825fec86e1ff38f9278423cd2f41ffca8e5212ea22aab7355fc7bd6d9" Jul 17 01:12:05.178218 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:12:05.178201 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9_odh-ai-gateway-infra(d0ae818f-c443-45bc-8bb0-b968deb2879d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" Jul 17 01:12:06.184496 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:06.184459 2572 generic.go:356] "Generic (PLEG): container finished" podID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerID="cc6008f5fbf43241ccfe34017e4bca8c8a40998ebc3df67362230a98a2f370cf" exitCode=6 Jul 17 01:12:06.184965 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:06.184544 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" event={"ID":"27b90b29-d87f-4d73-a80e-e771995f7d98","Type":"ContainerDied","Data":"cc6008f5fbf43241ccfe34017e4bca8c8a40998ebc3df67362230a98a2f370cf"} Jul 17 01:12:06.184965 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:06.184602 2572 scope.go:117] "RemoveContainer" containerID="c7dd3ca5e6feead448dabe7cb87a1cb186137288ab2c41609419b1622078ef3d" Jul 17 01:12:06.184965 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:06.184747 2572 scope.go:117] "RemoveContainer" containerID="cc6008f5fbf43241ccfe34017e4bca8c8a40998ebc3df67362230a98a2f370cf" Jul 17 01:12:06.185113 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:12:06.184977 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt_odh-ai-gateway-infra(27b90b29-d87f-4d73-a80e-e771995f7d98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" Jul 17 01:12:07.000625 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:07.000594 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:07.191746 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:07.191717 2572 scope.go:117] "RemoveContainer" containerID="cc6008f5fbf43241ccfe34017e4bca8c8a40998ebc3df67362230a98a2f370cf" Jul 17 01:12:07.192156 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:12:07.191914 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt_odh-ai-gateway-infra(27b90b29-d87f-4d73-a80e-e771995f7d98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" Jul 17 01:12:09.992459 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:09.992430 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:12:11.547007 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:11.546974 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:16.081956 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:16.081924 2572 scope.go:117] "RemoveContainer" containerID="255923224d1dea0505366251846547f5a10ea6ba89f44ba05735fc490ed47617" Jul 17 01:12:17.081753 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:17.081673 2572 scope.go:117] "RemoveContainer" containerID="ee2006f825fec86e1ff38f9278423cd2f41ffca8e5212ea22aab7355fc7bd6d9" Jul 17 01:12:17.248195 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:17.248167 2572 generic.go:356] "Generic (PLEG): container finished" podID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerID="3d61e0f0eef8b76e7268920ee2af862ff732296e8168d1457f29f80fdaa26623" exitCode=6 Jul 17 01:12:17.248561 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:17.248238 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" event={"ID":"745c2ea2-8c3e-466f-973e-d7536392cc1c","Type":"ContainerDied","Data":"3d61e0f0eef8b76e7268920ee2af862ff732296e8168d1457f29f80fdaa26623"} Jul 17 01:12:17.248561 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:17.248300 2572 scope.go:117] "RemoveContainer" containerID="255923224d1dea0505366251846547f5a10ea6ba89f44ba05735fc490ed47617" Jul 17 01:12:17.248669 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:17.248602 2572 scope.go:117] "RemoveContainer" containerID="3d61e0f0eef8b76e7268920ee2af862ff732296e8168d1457f29f80fdaa26623" Jul 17 01:12:17.248851 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:12:17.248832 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4692a03f-z242f_odh-ai-gateway-infra(745c2ea2-8c3e-466f-973e-d7536392cc1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" Jul 17 01:12:17.389545 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:17.389511 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:17.490254 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:17.490221 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:18.256779 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:18.256744 2572 generic.go:356] "Generic (PLEG): container finished" podID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerID="18d9b19fd518023e3516e07c116b91a4e3351abc6986fca301175927273853c9" exitCode=6 Jul 17 01:12:18.257226 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:18.256784 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" event={"ID":"d0ae818f-c443-45bc-8bb0-b968deb2879d","Type":"ContainerDied","Data":"18d9b19fd518023e3516e07c116b91a4e3351abc6986fca301175927273853c9"} Jul 17 01:12:18.257226 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:18.256836 2572 scope.go:117] "RemoveContainer" containerID="ee2006f825fec86e1ff38f9278423cd2f41ffca8e5212ea22aab7355fc7bd6d9" Jul 17 01:12:18.257226 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:18.257151 2572 scope.go:117] "RemoveContainer" containerID="18d9b19fd518023e3516e07c116b91a4e3351abc6986fca301175927273853c9" Jul 17 01:12:18.257416 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:12:18.257394 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9_odh-ai-gateway-infra(d0ae818f-c443-45bc-8bb0-b968deb2879d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" Jul 17 01:12:18.271389 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:18.271363 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f"] Jul 17 01:12:18.404911 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:18.404865 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" Jul 17 01:12:18.505490 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:18.505463 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745c2ea2-8c3e-466f-973e-d7536392cc1c-maas-api-service-ca\") pod \"745c2ea2-8c3e-466f-973e-d7536392cc1c\" (UID: \"745c2ea2-8c3e-466f-973e-d7536392cc1c\") " Jul 17 01:12:18.505788 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:18.505766 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745c2ea2-8c3e-466f-973e-d7536392cc1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "745c2ea2-8c3e-466f-973e-d7536392cc1c" (UID: "745c2ea2-8c3e-466f-973e-d7536392cc1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:12:18.606026 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:18.605998 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/745c2ea2-8c3e-466f-973e-d7536392cc1c-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:12:19.264343 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.264313 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" Jul 17 01:12:19.264343 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.264338 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f" event={"ID":"745c2ea2-8c3e-466f-973e-d7536392cc1c","Type":"ContainerDied","Data":"272cb1372a0b4c820b16dc79b9e9e972fc5988cb23748cacf331be4656aa6fa7"} Jul 17 01:12:19.264829 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.264402 2572 scope.go:117] "RemoveContainer" containerID="3d61e0f0eef8b76e7268920ee2af862ff732296e8168d1457f29f80fdaa26623" Jul 17 01:12:19.282923 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.282864 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9"] Jul 17 01:12:19.288797 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.288775 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f"] Jul 17 01:12:19.293839 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.293816 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4692a03f-z242f"] Jul 17 01:12:19.415414 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.415390 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" Jul 17 01:12:19.514769 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.514691 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae818f-c443-45bc-8bb0-b968deb2879d-maas-api-service-ca\") pod \"d0ae818f-c443-45bc-8bb0-b968deb2879d\" (UID: \"d0ae818f-c443-45bc-8bb0-b968deb2879d\") " Jul 17 01:12:19.515064 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.515043 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ae818f-c443-45bc-8bb0-b968deb2879d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0ae818f-c443-45bc-8bb0-b968deb2879d" (UID: "d0ae818f-c443-45bc-8bb0-b968deb2879d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:12:19.616335 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:19.616302 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ae818f-c443-45bc-8bb0-b968deb2879d-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:12:20.082559 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:20.082531 2572 scope.go:117] "RemoveContainer" containerID="cc6008f5fbf43241ccfe34017e4bca8c8a40998ebc3df67362230a98a2f370cf" Jul 17 01:12:20.088218 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:20.088190 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" path="/var/lib/kubelet/pods/745c2ea2-8c3e-466f-973e-d7536392cc1c/volumes" Jul 17 01:12:20.270676 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:20.270645 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" event={"ID":"d0ae818f-c443-45bc-8bb0-b968deb2879d","Type":"ContainerDied","Data":"393e6b0184e00c7231a6941946b323950c1eb02b97c706b3e6e6884dd7c39786"} Jul 17 01:12:20.271165 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:20.270706 2572 scope.go:117] "RemoveContainer" containerID="18d9b19fd518023e3516e07c116b91a4e3351abc6986fca301175927273853c9" Jul 17 01:12:20.271165 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:20.270652 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9" Jul 17 01:12:20.286433 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:20.286412 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9"] Jul 17 01:12:20.292310 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:20.292289 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-28705dd7-9ptz9"] Jul 17 01:12:21.277535 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:21.277502 2572 generic.go:356] "Generic (PLEG): container finished" podID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerID="b079eb52856fd277b771b4ed91e386462c2085c6bc2372618b96c70cc3b40077" exitCode=6 Jul 17 01:12:21.277989 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:21.277573 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" event={"ID":"27b90b29-d87f-4d73-a80e-e771995f7d98","Type":"ContainerDied","Data":"b079eb52856fd277b771b4ed91e386462c2085c6bc2372618b96c70cc3b40077"} Jul 17 01:12:21.277989 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:21.277630 2572 scope.go:117] "RemoveContainer" containerID="cc6008f5fbf43241ccfe34017e4bca8c8a40998ebc3df67362230a98a2f370cf" Jul 17 01:12:21.277989 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:21.277938 2572 scope.go:117] "RemoveContainer" containerID="b079eb52856fd277b771b4ed91e386462c2085c6bc2372618b96c70cc3b40077" Jul 17 01:12:21.278204 ip-10-0-135-32 kubenswrapper[2572]: E0717 01:12:21.278181 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt_odh-ai-gateway-infra(27b90b29-d87f-4d73-a80e-e771995f7d98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" Jul 17 01:12:22.088250 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:22.088217 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" path="/var/lib/kubelet/pods/d0ae818f-c443-45bc-8bb0-b968deb2879d/volumes" Jul 17 01:12:22.195154 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:22.195124 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:22.301074 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:22.301041 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt"] Jul 17 01:12:22.428801 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:22.428774 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:22.430398 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:22.430377 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" Jul 17 01:12:22.543157 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:22.543125 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b90b29-d87f-4d73-a80e-e771995f7d98-maas-api-service-ca\") pod \"27b90b29-d87f-4d73-a80e-e771995f7d98\" (UID: \"27b90b29-d87f-4d73-a80e-e771995f7d98\") " Jul 17 01:12:22.543536 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:22.543516 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b90b29-d87f-4d73-a80e-e771995f7d98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27b90b29-d87f-4d73-a80e-e771995f7d98" (UID: "27b90b29-d87f-4d73-a80e-e771995f7d98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:12:22.644416 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:22.644330 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b90b29-d87f-4d73-a80e-e771995f7d98-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Jul 17 01:12:23.045405 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:23.045309 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:23.147257 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:23.147217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:23.291017 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:23.290981 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" event={"ID":"27b90b29-d87f-4d73-a80e-e771995f7d98","Type":"ContainerDied","Data":"5e82e49b1d1f6326904a414d3059b368892789e32d694965c2aba24e9325c624"} Jul 17 01:12:23.291185 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:23.291039 2572 scope.go:117] "RemoveContainer" containerID="b079eb52856fd277b771b4ed91e386462c2085c6bc2372618b96c70cc3b40077" Jul 17 01:12:23.291185 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:23.290998 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt" Jul 17 01:12:23.313878 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:23.313849 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt"] Jul 17 01:12:23.318616 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:23.318592 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b7aa65f0-7h2rt"] Jul 17 01:12:24.087180 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:24.087146 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" path="/var/lib/kubelet/pods/27b90b29-d87f-4d73-a80e-e771995f7d98/volumes" Jul 17 01:12:32.972991 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:32.972905 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:36.287821 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:36.287783 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qhtjf"] Jul 17 01:12:37.629301 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:37.629247 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:43.566853 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:43.566814 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:43.666803 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:43.666765 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:48.286964 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:48.286932 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:48.511620 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:48.511582 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:12:54.899486 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:54.899460 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9c65dfff-nswv9_c2725a15-6cb6-4252-aa67-52c3aac340f0/copy-manifests/0.log" Jul 17 01:12:54.905430 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:54.905402 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9c65dfff-nswv9_c2725a15-6cb6-4252-aa67-52c3aac340f0/manager/0.log" Jul 17 01:12:55.889870 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:55.889844 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-b5ab5c-95vsj_540c3c74-d338-4d5e-a97c-491848138326/revoke-keys/0.log" Jul 17 01:12:56.614064 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:56.614033 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-e1dd24-kfm8q_d7143ca6-5d37-404d-bb13-442437242380/revoke-keys/0.log" Jul 17 01:12:58.684290 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:58.684261 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-qhtjf_70c2366c-7920-4156-8b71-e9484a72a488/limitador/0.log" Jul 17 01:12:58.798182 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:58.798151 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dcst9_b9575fa0-65d6-46c1-90ee-67f955c0b423/manager/0.log" Jul 17 01:12:59.456166 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:12:59.456115 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d679d599d-prk4d_d06e5708-5361-4e4c-88e2-2525d9001cf3/kube-auth-proxy/0.log" Jul 17 01:13:00.546929 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:00.546879 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxmqlm_b58a8f79-4a7b-42eb-bb97-fc3be9255b39/main/0.log" Jul 17 01:13:00.762442 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:00.762409 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-rrvfp_93b8ad88-1922-4533-bc52-9574420d058d/main/0.log" Jul 17 01:13:06.041145 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041106 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gf5nn/must-gather-5vnkc"] Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041471 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041483 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041496 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041506 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041520 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041528 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041536 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041543 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041551 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041556 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041562 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041567 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041574 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041579 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041593 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041603 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041655 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerName="revoke-keys" Jul 17 01:13:06.041655 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041664 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerName="revoke-keys" Jul 17 01:13:06.042261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041672 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerName="revoke-keys" Jul 17 01:13:06.042261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041680 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerName="revoke-keys" Jul 17 01:13:06.042261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041688 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerName="revoke-keys" Jul 17 01:13:06.042261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041695 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerName="revoke-keys" Jul 17 01:13:06.042261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041699 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerName="revoke-keys" Jul 17 01:13:06.042261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041762 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerName="revoke-keys" Jul 17 01:13:06.042261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041769 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="745c2ea2-8c3e-466f-973e-d7536392cc1c" containerName="revoke-keys" Jul 17 01:13:06.042261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041828 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b90b29-d87f-4d73-a80e-e771995f7d98" containerName="revoke-keys" Jul 17 01:13:06.042261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.041836 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ae818f-c443-45bc-8bb0-b968deb2879d" containerName="revoke-keys" Jul 17 01:13:06.044788 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.044770 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gf5nn/must-gather-5vnkc" Jul 17 01:13:06.047162 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.047138 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gf5nn\"/\"kube-root-ca.crt\"" Jul 17 01:13:06.047264 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.047203 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gf5nn\"/\"default-dockercfg-qjfmp\"" Jul 17 01:13:06.047264 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.047231 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gf5nn\"/\"openshift-service-ca.crt\"" Jul 17 01:13:06.053109 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.053089 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gf5nn/must-gather-5vnkc"] Jul 17 01:13:06.055050 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.055028 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:13:06.123117 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.123089 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd6zs\" (UniqueName: \"kubernetes.io/projected/9655701c-6461-46d1-a95a-6c8ccc54ab18-kube-api-access-qd6zs\") pod \"must-gather-5vnkc\" (UID: \"9655701c-6461-46d1-a95a-6c8ccc54ab18\") " pod="openshift-must-gather-gf5nn/must-gather-5vnkc" Jul 17 01:13:06.123277 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.123159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9655701c-6461-46d1-a95a-6c8ccc54ab18-must-gather-output\") pod \"must-gather-5vnkc\" (UID: \"9655701c-6461-46d1-a95a-6c8ccc54ab18\") " pod="openshift-must-gather-gf5nn/must-gather-5vnkc" Jul 17 01:13:06.224233 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.224200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9655701c-6461-46d1-a95a-6c8ccc54ab18-must-gather-output\") pod \"must-gather-5vnkc\" (UID: \"9655701c-6461-46d1-a95a-6c8ccc54ab18\") " pod="openshift-must-gather-gf5nn/must-gather-5vnkc" Jul 17 01:13:06.224379 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.224302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd6zs\" (UniqueName: \"kubernetes.io/projected/9655701c-6461-46d1-a95a-6c8ccc54ab18-kube-api-access-qd6zs\") pod \"must-gather-5vnkc\" (UID: \"9655701c-6461-46d1-a95a-6c8ccc54ab18\") " pod="openshift-must-gather-gf5nn/must-gather-5vnkc" Jul 17 01:13:06.224641 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.224619 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9655701c-6461-46d1-a95a-6c8ccc54ab18-must-gather-output\") pod \"must-gather-5vnkc\" (UID: \"9655701c-6461-46d1-a95a-6c8ccc54ab18\") " pod="openshift-must-gather-gf5nn/must-gather-5vnkc" Jul 17 01:13:06.233090 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.233064 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd6zs\" (UniqueName: \"kubernetes.io/projected/9655701c-6461-46d1-a95a-6c8ccc54ab18-kube-api-access-qd6zs\") pod \"must-gather-5vnkc\" (UID: \"9655701c-6461-46d1-a95a-6c8ccc54ab18\") " pod="openshift-must-gather-gf5nn/must-gather-5vnkc" Jul 17 01:13:06.245077 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.245052 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" limit=1024 Jul 17 01:13:06.355036 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.355012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gf5nn/must-gather-5vnkc" Jul 17 01:13:06.483079 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.483010 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gf5nn/must-gather-5vnkc"] Jul 17 01:13:06.485245 ip-10-0-135-32 kubenswrapper[2572]: W0717 01:13:06.485219 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9655701c_6461_46d1_a95a_6c8ccc54ab18.slice/crio-d3d707a7175f917050b852608e2d6bace65b4b718f7e1cd2ce4a40fd9cda318f WatchSource:0}: Error finding container d3d707a7175f917050b852608e2d6bace65b4b718f7e1cd2ce4a40fd9cda318f: Status 404 returned error can't find the container with id d3d707a7175f917050b852608e2d6bace65b4b718f7e1cd2ce4a40fd9cda318f Jul 17 01:13:06.503463 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:06.503437 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gf5nn/must-gather-5vnkc" event={"ID":"9655701c-6461-46d1-a95a-6c8ccc54ab18","Type":"ContainerStarted","Data":"d3d707a7175f917050b852608e2d6bace65b4b718f7e1cd2ce4a40fd9cda318f"} Jul 17 01:13:08.518931 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:08.518813 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gf5nn/must-gather-5vnkc" event={"ID":"9655701c-6461-46d1-a95a-6c8ccc54ab18","Type":"ContainerStarted","Data":"d5ae799815cc05b1e394a41dd1daed4ba5ad807bf82dabf1f60954361c29eb65"} Jul 17 01:13:08.518931 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:08.518870 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gf5nn/must-gather-5vnkc" event={"ID":"9655701c-6461-46d1-a95a-6c8ccc54ab18","Type":"ContainerStarted","Data":"04eb6357b78b1156ffd5c057af7de128c970f0f113e80c37b3a69d16696209b6"} Jul 17 01:13:09.111965 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:09.111852 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n78sr_dfabc064-bfdc-4cb0-923c-eb32d6ab392b/global-pull-secret-syncer/0.log" Jul 17 01:13:09.207095 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:09.207066 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m676v_1bd7118e-95e9-46b8-9e06-2579ca5607c3/konnectivity-agent/0.log" Jul 17 01:13:09.315121 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:09.315065 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-32.ec2.internal_9c41058aa23e327c16b34dda6898223f/haproxy/0.log" Jul 17 01:13:15.064060 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:15.064027 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-qhtjf_70c2366c-7920-4156-8b71-e9484a72a488/limitador/0.log" Jul 17 01:13:15.161711 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:15.161677 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dcst9_b9575fa0-65d6-46c1-90ee-67f955c0b423/manager/0.log" Jul 17 01:13:16.838830 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:16.838796 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22bb06a1-bac7-4bfc-b5b0-1efc004de6c5/alertmanager/0.log" Jul 17 01:13:16.866829 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:16.866803 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22bb06a1-bac7-4bfc-b5b0-1efc004de6c5/config-reloader/0.log" Jul 17 01:13:16.888389 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:16.888309 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22bb06a1-bac7-4bfc-b5b0-1efc004de6c5/kube-rbac-proxy-web/0.log" Jul 17 01:13:16.919627 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:16.919597 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22bb06a1-bac7-4bfc-b5b0-1efc004de6c5/kube-rbac-proxy/0.log" Jul 17 01:13:16.940851 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:16.940818 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22bb06a1-bac7-4bfc-b5b0-1efc004de6c5/kube-rbac-proxy-metric/0.log" Jul 17 01:13:16.960147 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:16.960107 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22bb06a1-bac7-4bfc-b5b0-1efc004de6c5/prom-label-proxy/0.log" Jul 17 01:13:16.982801 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:16.982773 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22bb06a1-bac7-4bfc-b5b0-1efc004de6c5/init-config-reloader/0.log" Jul 17 01:13:17.077097 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.077007 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vhr26_f896ad14-cb87-46cd-9eb9-13b10971743e/kube-state-metrics/0.log" Jul 17 01:13:17.093954 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.093926 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vhr26_f896ad14-cb87-46cd-9eb9-13b10971743e/kube-rbac-proxy-main/0.log" Jul 17 01:13:17.121531 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.121503 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vhr26_f896ad14-cb87-46cd-9eb9-13b10971743e/kube-rbac-proxy-self/0.log" Jul 17 01:13:17.361192 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.361161 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fm782_0b5ecc4a-f010-4fbe-95b7-792af5cef2f7/node-exporter/0.log" Jul 17 01:13:17.380262 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.380231 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fm782_0b5ecc4a-f010-4fbe-95b7-792af5cef2f7/kube-rbac-proxy/0.log" Jul 17 01:13:17.408310 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.408278 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fm782_0b5ecc4a-f010-4fbe-95b7-792af5cef2f7/init-textfile/0.log" Jul 17 01:13:17.451563 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.451529 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-hvrww_77419013-35d6-4961-8303-d0b49a55de56/kube-rbac-proxy-main/0.log" Jul 17 01:13:17.483183 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.483150 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-hvrww_77419013-35d6-4961-8303-d0b49a55de56/kube-rbac-proxy-self/0.log" Jul 17 01:13:17.504533 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.504507 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-hvrww_77419013-35d6-4961-8303-d0b49a55de56/openshift-state-metrics/0.log" Jul 17 01:13:17.546967 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.546936 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0068c298-8e29-472f-8e33-318812b54dbc/prometheus/0.log" Jul 17 01:13:17.565615 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.565574 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0068c298-8e29-472f-8e33-318812b54dbc/config-reloader/0.log" Jul 17 01:13:17.589677 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.589652 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0068c298-8e29-472f-8e33-318812b54dbc/thanos-sidecar/0.log" Jul 17 01:13:17.613723 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.613633 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0068c298-8e29-472f-8e33-318812b54dbc/kube-rbac-proxy-web/0.log" Jul 17 01:13:17.634090 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.634062 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0068c298-8e29-472f-8e33-318812b54dbc/kube-rbac-proxy/0.log" Jul 17 01:13:17.659431 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.659401 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0068c298-8e29-472f-8e33-318812b54dbc/kube-rbac-proxy-thanos/0.log" Jul 17 01:13:17.682703 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.682677 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0068c298-8e29-472f-8e33-318812b54dbc/init-config-reloader/0.log" Jul 17 01:13:17.761293 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.761264 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-glp6x_a0e59b55-7716-422c-a323-297e95c569bc/prometheus-operator-admission-webhook/0.log" Jul 17 01:13:17.785924 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.785881 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-669c5bb98c-t9nbj_281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c/telemeter-client/0.log" Jul 17 01:13:17.804691 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.804652 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-669c5bb98c-t9nbj_281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c/reload/0.log" Jul 17 01:13:17.826183 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.826119 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-669c5bb98c-t9nbj_281a2ab3-8cfa-42b5-b2b9-e4e5c72e806c/kube-rbac-proxy/0.log" Jul 17 01:13:17.855468 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.855437 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d89d8974-zdlqm_0a404540-3048-48c6-a355-b93bed008a7e/thanos-query/0.log" Jul 17 01:13:17.873358 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.873286 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d89d8974-zdlqm_0a404540-3048-48c6-a355-b93bed008a7e/kube-rbac-proxy-web/0.log" Jul 17 01:13:17.891441 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.891403 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d89d8974-zdlqm_0a404540-3048-48c6-a355-b93bed008a7e/kube-rbac-proxy/0.log" Jul 17 01:13:17.911476 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.911447 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d89d8974-zdlqm_0a404540-3048-48c6-a355-b93bed008a7e/prom-label-proxy/0.log" Jul 17 01:13:17.932108 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.932078 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d89d8974-zdlqm_0a404540-3048-48c6-a355-b93bed008a7e/kube-rbac-proxy-rules/0.log" Jul 17 01:13:17.949991 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:17.949954 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d89d8974-zdlqm_0a404540-3048-48c6-a355-b93bed008a7e/kube-rbac-proxy-metrics/0.log" Jul 17 01:13:19.469591 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.469517 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gf5nn/must-gather-5vnkc" podStartSLOduration=12.42041839 podStartE2EDuration="13.469491218s" totalImagesPullingTime="1.049072828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:13:08.535809172 +0000 UTC m=+3361.282318207" watchObservedRunningTime="2026-07-17 01:13:19.469491218 +0000 UTC m=+3372.216000253" Jul 17 01:13:19.470090 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.469806 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn"] Jul 17 01:13:19.476242 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.476215 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.483416 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.483383 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn"] Jul 17 01:13:19.565343 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.565303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tf7k\" (UniqueName: \"kubernetes.io/projected/8ddc1def-86dd-4efc-8d89-6e8f541e5435-kube-api-access-2tf7k\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.565522 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.565471 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-sys\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.565522 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.565514 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-lib-modules\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.565631 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.565586 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-podres\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.565672 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.565645 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-proc\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.666968 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.666917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-sys\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.667157 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.666975 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-lib-modules\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.667157 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.667022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-podres\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.667157 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.667063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-proc\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.667157 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.667127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf7k\" (UniqueName: \"kubernetes.io/projected/8ddc1def-86dd-4efc-8d89-6e8f541e5435-kube-api-access-2tf7k\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.667552 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.667527 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-lib-modules\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.667684 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.667663 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-sys\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.667833 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.667627 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-proc\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.667948 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.667587 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ddc1def-86dd-4efc-8d89-6e8f541e5435-podres\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.675144 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.675120 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tf7k\" (UniqueName: \"kubernetes.io/projected/8ddc1def-86dd-4efc-8d89-6e8f541e5435-kube-api-access-2tf7k\") pod \"perf-node-gather-daemonset-ncpfn\" (UID: \"8ddc1def-86dd-4efc-8d89-6e8f541e5435\") " pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.793229 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.793142 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:19.960592 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:19.960558 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn"] Jul 17 01:13:20.605726 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:20.605677 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" event={"ID":"8ddc1def-86dd-4efc-8d89-6e8f541e5435","Type":"ContainerStarted","Data":"53620ed7996b6413997409279b3d4a783a9bdf331ad8e0d49b06340af7ef5341"} Jul 17 01:13:20.606268 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:20.605753 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" event={"ID":"8ddc1def-86dd-4efc-8d89-6e8f541e5435","Type":"ContainerStarted","Data":"4267dba77d9bd23ef75966a79d15d84f5ad8129690716f240ee2a01a69b1f4b2"} Jul 17 01:13:20.606635 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:20.606574 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:20.621612 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:20.621558 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" podStartSLOduration=1.621538706 podStartE2EDuration="1.621538706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:13:20.620294714 +0000 UTC m=+3373.366803749" watchObservedRunningTime="2026-07-17 01:13:20.621538706 +0000 UTC m=+3373.368047741" Jul 17 01:13:21.890629 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:21.890588 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlvlj_6382649c-69a4-455a-9879-9e275333eb0f/dns/0.log" Jul 17 01:13:21.909467 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:21.909417 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlvlj_6382649c-69a4-455a-9879-9e275333eb0f/kube-rbac-proxy/0.log" Jul 17 01:13:22.026150 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:22.026120 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9tj4h_180b2640-39c5-4cf3-98c6-62deb22ebd32/dns-node-resolver/0.log" Jul 17 01:13:22.587537 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:22.587491 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mf9bv_1faae122-db93-4c37-bf65-16ed121ad6dc/node-ca/0.log" Jul 17 01:13:23.788780 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:23.788740 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d679d599d-prk4d_d06e5708-5361-4e4c-88e2-2525d9001cf3/kube-auth-proxy/0.log" Jul 17 01:13:24.686131 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:24.686100 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7wrvh_30a743cd-32ba-43eb-8805-991f080b23b4/serve-healthcheck-canary/0.log" Jul 17 01:13:25.206771 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:25.206735 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2kwdw_203fcf71-4593-418d-9bd1-f2e2876a071b/kube-rbac-proxy/0.log" Jul 17 01:13:25.227642 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:25.227610 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2kwdw_203fcf71-4593-418d-9bd1-f2e2876a071b/exporter/0.log" Jul 17 01:13:25.248461 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:25.248426 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2kwdw_203fcf71-4593-418d-9bd1-f2e2876a071b/extractor/0.log" Jul 17 01:13:27.627092 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:27.627060 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gf5nn/perf-node-gather-daemonset-ncpfn" Jul 17 01:13:28.125711 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:28.125677 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9c65dfff-nswv9_c2725a15-6cb6-4252-aa67-52c3aac340f0/manager/0.log" Jul 17 01:13:28.145656 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:28.145631 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9c65dfff-nswv9_c2725a15-6cb6-4252-aa67-52c3aac340f0/copy-manifests/0.log" Jul 17 01:13:29.357256 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:29.357218 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d74cbdbc7-jt9zt_443bb4e3-bb3b-4c05-a5a9-17559ebaf4e5/manager/0.log" Jul 17 01:13:29.381016 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:29.380985 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pkpfz_8a7d1aea-d729-4cb5-9f9a-04d40eadeed7/openshift-lws-operator/0.log" Jul 17 01:13:29.502854 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:29.502777 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d679d599d-prk4d_d06e5708-5361-4e4c-88e2-2525d9001cf3/kube-auth-proxy/0.log" Jul 17 01:13:31.249931 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:31.249856 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d679d599d-prk4d_d06e5708-5361-4e4c-88e2-2525d9001cf3/kube-auth-proxy/0.log" Jul 17 01:13:35.671872 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:35.671849 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8fbm4_6b254ab0-677e-4fd2-b0a2-ea3e76928696/kube-multus-additional-cni-plugins/0.log" Jul 17 01:13:35.692596 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:35.692570 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8fbm4_6b254ab0-677e-4fd2-b0a2-ea3e76928696/egress-router-binary-copy/0.log" Jul 17 01:13:35.709738 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:35.709714 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8fbm4_6b254ab0-677e-4fd2-b0a2-ea3e76928696/cni-plugins/0.log" Jul 17 01:13:35.726431 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:35.726407 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8fbm4_6b254ab0-677e-4fd2-b0a2-ea3e76928696/bond-cni-plugin/0.log" Jul 17 01:13:35.742669 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:35.742649 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8fbm4_6b254ab0-677e-4fd2-b0a2-ea3e76928696/routeoverride-cni/0.log" Jul 17 01:13:35.759119 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:35.759098 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8fbm4_6b254ab0-677e-4fd2-b0a2-ea3e76928696/whereabouts-cni-bincopy/0.log" Jul 17 01:13:35.776586 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:35.776563 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8fbm4_6b254ab0-677e-4fd2-b0a2-ea3e76928696/whereabouts-cni/0.log" Jul 17 01:13:36.169175 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:36.169142 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cx5p6_80c093d7-478e-4edf-820c-34720b639a43/kube-multus/0.log" Jul 17 01:13:36.229029 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:36.228993 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-75rcn_dc74d918-a560-4e0b-831f-c09d6a1826e9/network-metrics-daemon/0.log" Jul 17 01:13:36.244580 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:36.244553 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-75rcn_dc74d918-a560-4e0b-831f-c09d6a1826e9/kube-rbac-proxy/0.log" Jul 17 01:13:37.683271 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:37.683194 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qbd7_16fd2ba4-d022-4232-938c-ee9ab79b44f9/ovn-controller/0.log" Jul 17 01:13:37.699261 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:37.699233 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qbd7_16fd2ba4-d022-4232-938c-ee9ab79b44f9/ovn-acl-logging/0.log" Jul 17 01:13:37.720408 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:37.720375 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qbd7_16fd2ba4-d022-4232-938c-ee9ab79b44f9/kube-rbac-proxy-node/0.log" Jul 17 01:13:37.737882 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:37.737852 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qbd7_16fd2ba4-d022-4232-938c-ee9ab79b44f9/kube-rbac-proxy-ovn-metrics/0.log" Jul 17 01:13:37.755168 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:37.755139 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qbd7_16fd2ba4-d022-4232-938c-ee9ab79b44f9/northd/0.log" Jul 17 01:13:37.772844 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:37.772812 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qbd7_16fd2ba4-d022-4232-938c-ee9ab79b44f9/nbdb/0.log" Jul 17 01:13:37.789365 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:37.789334 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qbd7_16fd2ba4-d022-4232-938c-ee9ab79b44f9/sbdb/0.log" Jul 17 01:13:37.916391 ip-10-0-135-32 kubenswrapper[2572]: I0717 01:13:37.916351 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qbd7_16fd2ba4-d022-4232-938c-ee9ab79b44f9/ovnkube-controller/0.log"